💻 computer science

LumiTokens: 3D Relighting via Token-Space Lighting Transformation

LumiTokens is a novel 3D relighting framework that bypasses explicit material decomposition and rendering equations by directly transforming compact latent scene tokens into relit images through a self-attention-based editor, enabling unified, progressive, and composable lighting edits for diverse light sources.

Original authors: Yiwen Chen, Matheus Gadelha, Huaizu Jiang

Published 2026-08-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yiwen Chen, Matheus Gadelha, Huaizu Jiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine taking a photograph of a character in a movie studio or a product in a laboratory and wanting to place it into a completely different world. You might want that character to stand under the warm glow of a sunset or that product to sit on a kitchen counter lit by a morning window. For decades, computer graphics has struggled with this task. To make a 3D object look real in a new light, traditional methods required a painstaking process of reverse-engineering the object. Artists and algorithms had to peel back the layers of the image to figure out the object's shape, its texture, and how shiny or rough its surface was. Only after solving this complex puzzle could they reassemble the image under a new light. This approach was slow, often required hundreds of photos to work correctly, and was limited by how well the computer could guess the physical properties of the material.

A newer wave of technology has tried to solve this by using artificial intelligence to simply guess what the new image should look like, much like a painter improvising a scene. While these methods can create beautiful results, they often struggle to keep the object looking consistent when viewed from different angles. If you move your camera around the object, the lighting might look right from one side but wrong from another, breaking the illusion of a solid 3D world. Furthermore, changing the light usually meant starting the entire painting process over from scratch. There was no way to save the object's state and simply add a new lamp or move the sun without regenerating everything.

Researchers at Northeastern University and Adobe Research have introduced a new approach called LumiTokens that changes how we think about this problem. Instead of trying to figure out the physical shape of an object or guessing the final image pixel by pixel, their system treats the entire 3D scene as a compact set of digital tokens. Think of these tokens as a highly efficient, compressed summary of the object that contains all its geometry and appearance, but without being tied to any specific physical description like a 3D model or a grid of points. The researchers discovered that they could manipulate this summary directly. By feeding the system a description of a new light source, they could transform these tokens to reflect the new lighting conditions, and then decode the result into a fresh, high-quality image.

The core of this method is a tool they call a Scene Token Editor. This component takes the compressed summary of the scene and the description of the new light, and it blends them together using a mechanism similar to how a search engine connects related ideas. The system does not need to know the exact 3D coordinates of the object or the physics of how light bounces off a surface. Instead, it learns to adjust the tokens so that when they are turned back into an image, the shadows fall correctly, the highlights shine in the right places, and the reflections appear natural. Crucially, this process works for many different types of light, whether it is a broad sky, a single point bulb, or a large glowing panel. The system converts all these different light sources into a common language of light rays, allowing it to handle them uniformly.

One of the most significant capabilities of this new system is that it allows for progressive editing. Because the system works on the compressed summary rather than the final image, a user can build up a lighting scene one source at a time. They can start with a base environment, add a key light, and then add a fill light, with each step modifying the same underlying summary. The system remembers the changes made in the previous step, so the user does not need to start over or re-calculate the entire scene for every new addition. This creates a fluid workflow where lighting can be designed incrementally, much like adding ingredients to a dish, while maintaining a consistent look from every angle.

The researchers tested their system on thousands of 3D objects, ranging from simple shapes to complex models with shiny and transparent surfaces. They found that their method produced images that were as sharp and accurate as the best existing techniques, and in many cases, superior. When they compared their results to other methods that rely on guessing the final image or those that try to reverse-engineer the physics, LumiTokens showed fewer errors in how the light interacted with the object. It avoided common mistakes like inconsistent shadows or lighting that looked different depending on the camera angle. The system was able to take just a handful of photos of an object and relight it under conditions it had never seen before, producing results that looked physically plausible.

This work suggests a new path forward for creating 3D assets for movies, video games, and virtual reality. By moving away from the need to explicitly define physical materials or run heavy physics simulations, the researchers have shown that lighting can be treated as a direct transformation of a scene's digital essence. The system does not claim to understand the physics of light in the way a human physicist does, but it has learned to mimic the visual outcome so effectively that the difference is invisible to the eye. This approach offers a faster, more flexible way to bring digital objects into new worlds, allowing creators to adjust the mood and atmosphere of a scene with a level of control that was previously difficult to achieve.

Technical Summary: LumiTokens: 3D Relighting via Token-Space Lighting Transformation

Problem Statement

3D relighting—the ability to render a reconstructed scene under arbitrary target illumination—is a critical step for deploying 3D assets in practical applications such as film production, video games, and e-commerce. Existing approaches to this problem generally fall into three categories, each with significant trade-offs:

Inverse Rendering: These methods recover explicit 3D representations (geometry, normals, albedo, roughness) and re-evaluate rendering equations. While multi-view consistent, they require dense captures, costly per-scene optimization, and are limited by the expressiveness of their chosen BRDF models.
Diffusion-Based Generation: These methods leverage diffusion models as appearance priors to synthesize relit views. While they produce realistic effects, they typically operate in view-space or video-space without a shared, persistent scene representation. Consequently, each lighting change requires a full generative pass, and lighting edits do not accumulate.
Hybrid Approaches: These combine feed-forward geometry encoding with generative appearance synthesis. While they avoid per-scene optimization, they still rely on explicit 3D parameterizations (like 3DGS) and generative branches that must be re-run for every new lighting condition.

A common limitation across all existing paradigms is that lighting edits are not maintained as incremental updates to a persistent scene representation. This paper observes that recent latent scene representations, which encode multi-view images into compact tokens without fixed physical semantics, offer a novel design space where relighting can be formulated as a direct transformation in latent space.

Methodology

The authors propose LumiTokens, a transformer-based framework that performs 3D relighting entirely within a learned latent token space, bypassing explicit 3D representations, rendering equations, and physics-based decomposition.

Architecture Overview

The framework consists of three main components:

Encoder: A transformer that maps a sparse set of posed multi-view input images (and their Plücker ray embeddings) into a compact, fixed-size set of 1D unstructured scene tokens ( $z$ ).
Scene Token Editor: A transformer module that performs the relighting operation. It takes the scene tokens ( $z$ ) and a set of light-ray tokens ( $\ell$ ) as input. Through self-attention, the editor jointly processes scene and light tokens to produce updated scene tokens ( $z'$ ). Crucially, the output remains in the same latent space as the input, allowing for persistent editing.
Decoder: A ray-conditioned transformer that renders novel views from the scene tokens. It uses a DPT head to reassemble output tokens into dense, high-resolution images, ensuring sharp textures and high-frequency details.

Light Signal Tokenization

To support diverse lighting types through a unified interface, all lighting signals are parameterized as Plücker ray tokens. The illumination is discretized into a collection of light rays sampled from the source toward the scene. Each light token is defined by its ray geometry (Plücker coordinates), spectral intensity, and radiant power. This unified representation accommodates:

Environment Maps: Modeled as directional rays with zero moment.
Point Lights: Modeled as rays originating from a specific 3D position.
Area Lights: Modeled as rays sampled across an emitting surface.

This tokenization enables native 3D user interaction (e.g., placing a point light at a specific coordinate) despite the latent space lacking explicit spatial structure.

Progressive Relighting

A key feature of LumiTokens is progressive relighting. Because the editor's output ( $z'$ ) remains in the same latent space as its input ( $z$ ), users can incrementally build up illumination one light source at a time. The editing process follows a recurrence:
$z^{(s)} = T(z^{(s-1)}, \ell^{(s)})$
where $z^{(s)}$ is the scene state after $s$ edits. This allows lighting edits to compose in token space without requiring re-encoding, re-evaluation of rendering equations, or regeneration of views at each step.

Training Objectives

The model is trained by optimizing the encoder and the Scene Token Editor jointly, while keeping a pre-trained decoder frozen. The training loss combines:

Rendering Loss ( $\mathcal{L}_{render}$ ): A combination of pixel-wise $\ell_2$ loss and LPIPS perceptual loss, supervising the decoded output against ground-truth relit images. To support progressive editing, the model is trained on random chains of lighting signals, applying the loss at each step of the sequence to prevent distributional drift.
Lighting Invariance Loss ( $\mathcal{L}_{inv}$ ): This loss penalizes differences in encoded tokens when the same scene is captured under different source illuminations. This encourages the encoder to factor out source lighting effects, ensuring scene tokens capture persistent identity (geometry and materials) while the editor handles lighting-dependent variations.

Key Contributions

Latent Token Relighting: The paper demonstrates that 3D relighting can be performed as a direct transformation in a learned latent token space, eliminating the need for explicit 3D representations, rendering equations, or physics-based decomposition.
Scene Token Editor: The introduction of a transformer module that performs relighting via self-attention between scene tokens and Plücker ray tokens. This supports a unified interface for environment maps, point lights, and area lights, enabling native 3D user interaction.
Progressive and Composable Editing: The formulation supports progressive relighting where lighting edits compose in token space. This enables incremental lighting design, a capability existing paradigms can only achieve through full recomputation at each step.

Experimental Results

The authors evaluate LumiTokens on a large-scale synthetic dataset (20k objects from Objaverse) and compare against state-of-the-art methods.

Multi-view Relighting: LumiTokens achieves superior quantitative performance compared to diffusion-based baselines (Neural Gaffer, DiffusionRenderer, LightSwitch) in terms of PSNR, SSIM, and LPIPS. It preserves appearance and lighting effects consistently across views, whereas baselines exhibit artifacts like residual source illumination and inconsistent highlights.
Novel-view Relighting: When evaluated on unseen viewpoints, LumiTokens outperforms inverse rendering methods (TensoIR, NVDiffrecMC) and the multi-view diffusion method (LightSwitch). Notably, LumiTokens achieves these results using significantly fewer input views (8) compared to the inverse rendering baselines (50 views) and the diffusion baseline LightSwitch (16 views).
Qualitative Performance: Visual comparisons show that LumiTokens successfully handles complex lighting scenarios, including inter-object shadows and reflections, and supports the progressive addition of light sources (e.g., adding an environment map, then a point light, then an area light) with high fidelity.

Significance and Claims

The paper claims that LumiTokens represents a fundamental shift in relighting paradigms. By treating relighting as a transformation on latent scene tokens rather than a decomposition or generation problem, the method achieves:

Persistence: Lighting edits accumulate in a reusable scene state without regeneration.
Efficiency: It avoids the computational cost of per-scene optimization or full diffusion passes for every lighting change.
Flexibility: It supports diverse lighting types and progressive editing through a unified token interface.

The authors note that while prior work has used latent tokens for view synthesis, this is the first work to demonstrate that such representations can support relighting through direct, composable transformations in token space. The approach addresses the challenge of maintaining coherent lighting transformations in a space where geometry and appearance are entangled, enabling a new form of interactive 3D editing.