Light of Normals: Unified Feature Representation for Universal Photometric Stereo

This paper introduces LINO UniPS, a unified photometric stereo framework that achieves state-of-the-art performance under arbitrary lighting by employing Light Register Tokens and Interleaved Attention for robust illumination-normal decoupling, alongside a Wavelet-based Dual-branch Architecture to preserve high-frequency geometric details, all trained on the newly proposed large-scale PS-Verse dataset.

The Gist

Imagine you are trying to figure out the shape of a mysterious object in a dark room. You have a flashlight, but you can't move it; instead, you have a friend who shines the light on the object from 16 different angles. Your goal is to build a 3D mental model of the object just by looking at how the shadows and highlights shift.

This is the challenge of Photometric Stereo. For a long time, computers were terrible at this unless the lighting was perfectly controlled (like in a lab). If the light was weird or the object was shiny, the computer got confused.

This paper introduces a new AI system called LINO UniPS (which stands for "Light of Normals"). Think of it as a super-smart detective that can look at those 16 photos and instantly build a perfect 3D map, even if the lighting is chaotic.

Here is how it works, explained with some everyday analogies:

1. The Problem: The "Confused Chef"

Previous AI models were like a chef trying to bake a cake while someone kept changing the oven temperature and throwing different ingredients into the mix. The chef (the AI) had to guess which part of the cake was the "flour" (the shape) and which part was the "heat" (the lighting). Because the chef couldn't separate the two, the final cake (the 3D shape) often looked mushy or had the wrong texture.

2. The Solution: The "Specialized Note-Takers" (Light Register Tokens)

The authors realized the AI needed a way to separate the "light" from the "shape" immediately.

• The Analogy: Imagine you are at a noisy party. You want to hear a specific conversation, but there are three types of noise: a bass drum (Point lights), a wind chime (Directional lights), and the general hum of the crowd (Environment lights).
• The Fix: LINO gives the AI three special "note-takers" (called Light Register Tokens).
  • One note-taker only writes down the bass drum sounds.
  • One only writes down the wind chimes.
  • One only writes down the crowd noise.
• The Result: By having these specialized note-takers, the AI can say, "Okay, I know exactly what the noise is. Now, let's ignore the noise and focus purely on the shape of the object." This is called decoupling.

3. The "Global Detective" (Interleaved Attention)

Once the AI has separated the noise from the signal, it needs to put the puzzle pieces together.

• The Analogy: Imagine looking at a jigsaw puzzle where the pieces are scattered across different tables. Old AI models looked at one table at a time. LINO uses a Global Detective that can see all the tables at once. It connects the dots between all 16 photos simultaneously, ensuring the final picture is consistent and doesn't have "glitches" where the lighting changes.

4. Keeping the Details: The "Wavelet Microscope"

One of the biggest problems with 3D reconstruction is losing fine details. If you take a photo and shrink it to make it easier to process, you lose the tiny scratches and textures.

• The Analogy: Imagine trying to describe a detailed embroidery pattern. If you just take a blurry photo of it, you miss the tiny stitches.
• The Fix: LINO uses a Wavelet-based Dual-branch Architecture. Think of this as having two pairs of eyes:
  • Eye 1 (The Downsample): Looks at the big picture to understand the overall shape (like seeing the whole flower).
  • Eye 2 (The Wavelet): Uses a "microscope" to look at the high-frequency details (the tiny stitches) that usually get lost when images are shrunk.
• The Result: The AI doesn't just guess the shape; it preserves the tiny, sharp details, making the 3D model look incredibly realistic.

5. The Training Ground: "PS-Verse"

To teach this AI, the authors didn't just use a few photos. They built a massive, virtual universe called PS-Verse.

• The Analogy: Instead of showing a student 10 math problems, they gave them 100,000 problems, starting with easy ones (smooth balls) and gradually getting harder (complex, bumpy rocks with shiny surfaces).
• Curriculum Learning: The AI learned like a human student: "First, let's master the easy shapes. Now, let's try the tricky ones." This made the AI incredibly good at handling real-world objects it had never seen before.

Why is this a big deal?

• It's Sharper: The 3D models it creates look almost as good as if you had scanned the object with a $10,000 industrial 3D scanner.
• It's Faster: It can process high-resolution images in seconds, whereas previous methods took minutes or hours.
• It's Universal: It works on anything—shiny metal, matte clay, or complex fabrics—without needing to be re-tuned for each object.

In short: LINO UniPS is like giving a computer a pair of noise-canceling headphones (to ignore the lighting) and a high-powered microscope (to see the tiny details), allowing it to see the true 3D shape of the world, no matter how the lights are arranged.

Technical Summary

Here is a detailed technical summary of the paper "LIGHT OF NORMALS: UNIFIED FEATURE REPRESENTATION FOR UNIVERSAL PHOTOMETRIC STEREO" (LINO UniPS).

1. Problem Statement

Universal Photometric Stereo (PS) aims to recover surface normal maps from a set of images taken under varying, unknown lighting conditions without relying on pre-calibrated light sources or specific physical illumination models.

Despite recent progress (e.g., SDM UniPS, Uni MS-PS), the field faces two critical challenges:

1. Ineffective Decoupling of Illumination and Normals: Existing encoders often process lighting and surface geometry jointly. Without explicitly separating (decoupling) the illumination cues from the normal cues, the decoder receives unstable features, leading to inconsistent normal predictions.
2. Loss of High-Frequency Geometric Details: Conventional downsampling operations (e.g., bilinear interpolation, pixel shuffling) used in encoder-decoder architectures tend to smooth out fine details or disrupt high-frequency semantics. This results in poor reconstruction of intricate surface textures and complex geometries.

2. Methodology: LINO UniPS

The authors propose LINO UniPS, a Vision Transformer (ViT)-based framework designed to explicitly decouple illumination from surface cues while preserving geometric detail. The architecture consists of three main stages:

A. Wavelet Feature Extractor (Addressing Detail Loss)

To prevent the loss of high-frequency information during downsampling, the input multi-light images are processed through a Dual-branch Architecture:

• Downsample Branch: Standard downsampling to capture global semantic context.
• Wavelet Branch: Uses Discrete Wavelet Transform (DWT) to decompose images into low-frequency (LL) and high-frequency (LH, HL, HH) components. This allows the network to explicitly attend to fine-grained geometric details.
• Fusion: Features from both branches are fused later in the network, and an Inverse Wavelet Transform (IDWT) is used during upsampling to reconstruct high-frequency details.

B. Light Registered Attention Module (Addressing Decoupling)

This module introduces a novel mechanism to explicitly represent and separate lighting information:

• Light Register Tokens: Inspired by "register" mechanisms in DINO, the authors introduce learnable tokens specifically for three types of lighting: Point, Directional, and Environment (Env) lights.
• Light Alignment Supervision: Unlike unsupervised registers, these tokens are trained with an explicit Light Alignment Loss. The tokens are forced to align with the ground-truth lighting parameters (position, intensity, HDRI maps) used during rendering. This ensures the tokens capture pure illumination characteristics.
• Interleaved Attention Block: The image features and the specialized Light Register Tokens are processed together using a global cross-image attention mechanism. The block interleaves four attention types: Frame → Light → Global → Light. The Global Attention layer allows the model to aggregate information across all input images simultaneously, enabling the encoder to factor out lighting variations while retaining normal-related evidence.

C. Normal-Gradient Perception Loss

To further refine the reconstruction of fine details, a specialized loss function is introduced:

• It generates a confidence map based on the predicted normal gradients.
• It heavily penalizes errors in high-frequency regions (areas with complex geometry) during training, forcing the network to prioritize the recovery of intricate surface details.

3. Key Contributions

1. Unified Feature Representation: The introduction of Light Register Tokens combined with Interleaved Attention and Light Alignment supervision successfully decouples illumination from surface normals, creating a unified feature space where lighting is explicitly represented by tokens and geometry is preserved in the feature map.
2. High-Frequency Preservation: The Wavelet-based Dual-branch Architecture and Normal-gradient Perception Loss significantly improve the reconstruction of fine-grained geometric details that are typically lost in standard PS methods.
3. PS-Verse Dataset: The authors constructed PS-Verse, a large-scale synthetic dataset (100,000 scenes) graded by geometric complexity (Levels 1–5) and lighting diversity. Crucially, it includes Level 5 scenes rendered with Normal Mapping to simulate high-frequency surface details, a feature missing in previous datasets.
4. Curriculum Training: The model is trained using a curriculum strategy, progressing from simple (Level 1) to complex (Level 5) scenes, enhancing generalization.

4. Experimental Results

The method was evaluated on public benchmarks (DiLiGenT, Luces) and the new PS-Verse Testdata.

• State-of-the-Art Performance:
  • DiLiGenT: Achieved an average Mean Angular Error (MAE) of 4.65°, outperforming previous bests (SDM UniPS: 5.80°, Uni MS-PS: 5.01°).
  • Luces: Achieved an average MAE of 9.43°, a significant improvement over the prior best (11.21°).
• Feature Consistency: The method demonstrated the highest Cosine Similarity (CSIM) and Structural Similarity (SSIM) among encoder features, confirming that the proposed decoupling mechanism yields more consistent, illumination-invariant features.
• Efficiency: LINO UniPS is significantly faster than Uni MS-PS (which requires ~35x more computation time) and slightly faster than SDM UniPS, making it suitable for high-resolution (4K) processing.
• Ablation Studies:
  • Removing Light Register Tokens or Global Attention degraded performance, proving their necessity for decoupling.
  • Removing the Wavelet branch or Gradient Loss resulted in smoother, less detailed normals.
  • The "Ours w/ MLP" variant (using a simple MLP decoder) still outperformed SDM UniPS, proving that the encoder's superior feature extraction is the primary driver of performance.

5. Significance

• Theoretical Advancement: The paper shifts the paradigm of Universal PS from "learning to disentangle" (which is difficult for decoders) to "explicitly disentangling" at the encoder level using specialized tokens and supervision.
• Practical Utility: By preserving high-frequency details and handling complex, real-world lighting (including point sources and environment maps), LINO UniPS bridges the gap between synthetic training and real-world application, producing results that visually rival 3D scanners.
• Resource Contribution: The release of PS-Verse provides the community with a rigorous benchmark for training models on complex geometries and diverse lighting, addressing a major bottleneck in data-driven Universal PS.

In summary, LINO UniPS establishes a new state-of-the-art by treating illumination and geometry as distinct entities within a unified transformer architecture, supported by a novel dataset and training curriculum.