💻 computer science

VENI: Variational Encoder for Natural Illumination

The paper proposes VENI, a rotation-equivariant variational autoencoder that utilizes a novel Vector Neuron Vision Transformer and a conditional neural field to model natural illumination on a sphere without 2D projections, thereby achieving a well-behaved latent space with smoother interpolation compared to existing methods.

Original authors: Paul Walker, James A. D. Gardner, Andreea Ardelean, William A. P. Smith, Bernhard Egger

Published 2026-06-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Paul Walker, James A. D. Gardner, Andreea Ardelean, William A. P. Smith, Bernhard Egger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to reverse-engineer a photograph. You have a picture of a scene, but you don't know what the sun looked like, where the clouds were, or what time of day it was. This is called "inverse rendering," and it's a tricky puzzle because many different lighting setups could create the exact same picture.

To solve this, computers usually need a "rulebook" or a prior—a set of learned expectations about how natural light usually behaves. For example, we know the sun is usually up, not down, and that light comes from a limited range of colors.

The paper introduces a new tool called VENI (Variational Encoder for Natural Illumination). Here is how it works, explained through simple analogies:

The Problem with the Old Way (RENI++)

Before VENI, the best tool for this job was called RENI++. Think of RENI++ like a blind sculptor trying to recreate a statue based on a photo.

The Issue: Every time the sculptor starts a new statue (a new image), they pick up a random lump of clay (a random "latent code") and start chiseling.
The Flaw: Because they start with a random lump every time, two very similar photos might end up with completely different lumps of clay. Worse, the sculptor might accidentally make the same statue look like two totally different lumps of clay. This makes the "clay library" messy and confusing. If you try to blend two lumps of clay together to make a new statue, the result might be a weird, broken mess.

The VENI Solution: A Smart, Organized Library

VENI changes the game by acting like a smart librarian instead of a blind sculptor.

The Encoder (The Librarian): When VENI sees a photo, it doesn't guess. It has a special system that instantly looks at the picture and finds the exact right "clump of clay" (latent code) from its organized library to represent that specific lighting.
The Decoder (The Sculptor): Once it has the right clump of clay, it uses a special 3D printer (a neural field) to recreate the lighting environment perfectly.

The Secret Sauce: Rotation and 3D Thinking

The paper highlights two main tricks VENI uses to be better than the competition:

1. The "Spinning Top" Rule (Rotation-Equivariance)
Imagine a spinning top. If you spin it, the top looks different from the side, but it's still the same top. Natural lighting works the same way. If you rotate a room 90 degrees, the lighting changes, but the rules of how the light behaves stay the same.

Old models often had to take a 3D sphere of light, squash it flat onto a 2D piece of paper (like a map), and then try to learn from it. This squashing creates distortions (like how Greenland looks huge on a flat map).
VENI works directly in 3D space. It treats the light like a spinning top. It uses a special "Vector Neuron" brain that understands that if you rotate the input, the output should just rotate too, without getting confused or distorted. It's like learning to dance by actually spinning, rather than trying to learn dance moves from a flat drawing.

2. The "Magic Compass" (SO(2)-Equivariance)
The authors realized that while light can spin around the vertical axis (like a compass needle), it doesn't make sense for light to flip upside down or tilt sideways in a natural outdoor scene.

They built a special layer in their AI that is smart enough to know: "Okay, spinning left and right is fine, but don't let the colors or the 'up' direction get messed up." This makes the model much more efficient and accurate.

Why VENI is Better

The paper proves VENI wins in two main ways:

A Neat Library (Uniqueness): In the old method, two similar photos might get two totally different "clay codes." In VENI, two similar photos get two very similar codes. This means the "library" is organized. If you look up a code, you know exactly what lighting it represents.
Smooth Blending (Interpolation): Because the library is organized, you can take the code for a "Sunrise" and the code for "Noon" and blend them together.
- Old Method: Blending them might create a weird artifact, like a sun appearing out of nowhere in the middle of the sky.
- VENI: Blending them creates a smooth, realistic transition where the sun moves naturally across the sky and the colors shift gently from orange to blue.

Summary

VENI is a new AI system that learns how natural light works by treating it as a 3D object that can spin, rather than a flat, distorted image. It organizes its knowledge so that similar lights have similar "IDs," allowing it to smoothly transition between different times of day without creating weird visual glitches. It's a more reliable, organized, and mathematically sound way for computers to understand the lighting in our world.

Technical Summary: VENI: Variational Encoder for Natural Illumination

Problem Statement

Inverse rendering is an ill-posed problem, as a single image can be generated by multiple combinations of shape, material, and lighting. While humans utilize strong learned priors over illumination spaces to solve this, existing computational approaches often fail to adequately model natural illumination. Current state-of-the-art methods face two primary limitations:

Lack of Rotation-Equivariance: Many models disregard the spherical and rotation-equivariant nature of illumination environments. While some recent works (e.g., RENI++) attempt to address this, they often rely on 2D projections (equirectangular or cube maps) which introduce distortion, or they fail to maintain equivariance in the encoder.
Poor Latent Space Properties: Existing rotation-equivariant priors, such as RENI++, utilize an "autodecoder" architecture. In this setup, latent codes are randomly initialized for each image and jointly optimized with the model. This leads to a non-unique latent space where similar images may be represented by vastly different latent codes. Furthermore, this approach does not scale to large datasets because a new latent code must be optimized for every single image.

Methodology

The authors propose VENI (Variational Encoder for Natural Illumination), a rotation-equivariant variational autoencoder (VAE) designed to model natural illumination on the sphere without relying on 2D projections. The architecture consists of three main components:

1. SO(2)-Equivariant Fully Connected Layer

To handle the specific symmetries of natural outdoor illumination, the authors extend the Vector Neuron framework. While standard Vector Neurons provide SO(3)-equivariance, natural illumination environments possess a canonical "up" direction, making only rotations around the vertical axis (SO(2)) physically meaningful. The authors design a novel SO(2)-equivariant fully connected layer that:

Separates input into equivariant components (x, y dimensions) and invariant components (z dimension and color).
Combines invariant and equivariant operations within a single neuron.
Ensures that the invariant dimensions (color, vertical axis) remain invariant to xy-plane rotations while still influencing the equivariant dimensions.
This layer is shown to outperform standard Vector Neurons when applied within an SO(2)-equivariant model.

2. Vector Neuron Vision Transformer (VN-ViT) Encoder

The core of the encoder is a Vision Transformer adapted for spherical data and rotation-equivariance:

Input Representation: Instead of 2D projections, the model operates directly in the 3D domain. Patches are sampled as vertical stripes on the sphere. To maintain equivariance without positional encoding (which would break it), color values are concatenated with the 3D direction vectors of the sampled pixels.
Architecture: All standard ViT components are replaced with their Vector Neuron counterparts. A learnable "class token" is prepended to the patch embeddings; this token is learnable only in invariant dimensions to preserve equivariance.
Output: The transformer output is projected via two SO(2)-equivariant fully connected layers to produce the mean ( $\mu$ ) and variance ( $\sigma$ ) for the latent code.

3. Rotation-Equivariant Variational Sampling and Decoder

Sampling: To ensure the latent space is rotation-equivariant, the authors sample from a 3D isotropic spherical normal distribution defined by a 3D mean vector and a 1D variance, rather than sampling three independent 1D distributions.
Decoder: The model adopts the RENI++ decoder, a rotation-equivariant conditional neural field. It takes a direction vector and the 3D latent vector as input to output color. The conditioning is achieved by encoding the direction relative to the latent code and using Vector Neuron invariant layers.

Training Strategy

The model is trained on a combination of datasets to handle the full dynamic range of natural light:

RENI++ Dataset: 1,694 high-quality HDR equirectangular images (outdoor scenes).
StreetLearn Dataset: 43,310 LDR 360° images from Google Street View, converted to HDR.
Curriculum Learning: The model is first pretrained on the large StreetLearn dataset and then fine-tuned on the smaller, higher-quality RENI++ dataset.
Loss Functions: The training utilizes a log-space loss to match human visual perception, a Mean Absolute Gradient Error (MAGE) to capture high-frequency details, a scale-invariant loss, a cosine similarity loss for color accuracy, and a Kullback-Leibler divergence (KLD) term adjusted for 3D isotropic distributions.

Key Contributions

Novel Encoder Architecture: A rotation-equivariant panoramic vision transformer (VN-ViT) enabled by a new SO(2)-equivariant extension to Vector Neurons, allowing direct processing of spherical signals without 2D projection distortion.
Well-Behaved Latent Space: A rotation-equivariant natural illumination prior that produces a unique latent space, overcoming the non-uniqueness issues of autodecoder-based approaches like RENI++.
Scalability: By encoding images via a forward pass rather than optimizing latent codes per image, the method scales to large datasets.
Comprehensive Evaluation: Extensive evaluation using intuitive metrics for latent space uniqueness and reconstruction consistency.

Results

The paper evaluates VENI against RENI++ across various latent dimensions ( $D=27, 147, 300$ ):

Reconstruction Quality: VENI outperforms RENI++, particularly in low-dimensional settings ( $D=27$ ), achieving higher PSNR, SSIM, and lower LPIPS scores in both LDR tone-mapped and linear HDR spaces.
Latent Space Uniqueness: The authors demonstrate that VENI's latent space is significantly more unique. When two latent codes are optimized for the same image, the interpolation between them yields consistent results for VENI, whereas RENI++ shows significant divergence (MSE of 0.04 vs 1.46 for $D=27$ ).
Interpolation: Qualitative results show that VENI enables smooth, semantically meaningful interpolations (e.g., transitioning from sunrise to midday) without introducing artifacts like spurious light sources or noise, which are observed in RENI++ interpolations.
Scalability: Experiments show that VENI maintains performance when trained on the large StreetLearn dataset, whereas RENI++ (which requires per-image optimization) cannot scale to such datasets effectively.

Significance

The paper claims that VENI addresses critical limitations in current natural illumination priors by combining rotation-equivariance with a well-structured, unique latent space. By moving away from the autodecoder paradigm to a variational encoder, the model ensures that similar images map to similar latent codes, facilitating smooth interpolation and enabling the use of large-scale datasets. This results in a more robust prior for inverse rendering tasks, capable of handling the complex statistical regularities of natural outdoor illumination while respecting the physical constraints of spherical geometry.