💻 computer science

ICTPolarReal: A Polarized Reflection and Material Dataset of Real World Objects

This paper introduces ICTPolarReal, a large-scale dataset of over 1.2 million high-resolution images capturing 218 real-world objects under diverse multiview, multi-illumination, and polarization conditions, which significantly advances the accuracy and generalization of inverse rendering tasks like intrinsic decomposition, relighting, and 3D reconstruction by providing physically grounded material data.

Original authors: Jing Yang, Krithika Dharanikota, Emily Jia, Haiwei Chen, Yajie Zhao

Published 2026-03-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jing Yang, Krithika Dharanikota, Emily Jia, Haiwei Chen, Yajie Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to understand the physical world just by looking at photos. The robot sees a shiny red apple and a matte red brick. To a human, it's obvious: the apple is smooth and shiny (specular), while the brick is rough and dull (diffuse). But to a computer, they both just look like "red pixels."

This is the core problem the paper solves: How do we teach computers to see the real physics of light and materials, not just the colors?

Here is a simple breakdown of what the researchers did, using some everyday analogies.

1. The Problem: The "Plastic World" of AI

Currently, most AI models that understand 3D shapes and materials are trained on synthetic data (computer-generated images).

The Analogy: Imagine trying to learn how to cook a perfect steak by only watching cartoons of people cooking. You might learn the steps, but you won't understand the sizzle, the smell, or how the meat actually changes texture when it hits the heat.
The Issue: Computer graphics are great at making things look pretty, but they often simplify how light bounces off things. They ignore complex real-world physics like polarization (how light waves vibrate) and multi-layered reflections. Because of this, AI trained on these "cartoons" fails when it sees a real photo of a messy kitchen or a shiny car.

2. The Solution: The "Super-Light Stage"

The researchers built a massive, high-tech capture room called a Light Stage.

The Setup: Imagine a giant geodesic dome (like a futuristic igloo) covered in 346 tiny LED lights. Inside, there are 8 high-definition cameras circling the object.
The Magic Trick (Polarization): This is the secret sauce. The lights and cameras have special sunglasses (polarizers) on them.
- Parallel Sunglasses: Let both the "shiny" reflection and the "matte" color through.
- Crossed Sunglasses: Block the "shiny" reflection, letting only the "matte" color through.
The Result: By taking pictures with these different "sunglasses," the system can mathematically peel apart the image. It separates the "shiny highlight" from the "true color" of the object, just like peeling an orange to get to the fruit inside.

3. The Dataset: The "Ultimate Library of Objects"

They didn't just scan one thing; they scanned 218 everyday objects.

What's in the box? Apples, bananas, metal kettles, glass bottles, rubber balls, fabric bags, and even marbles.
The Scale: They took over 1.2 million photos of these objects.
- Every object was lit from 346 different angles.
- Every object was viewed from 8 different angles.
- Every photo was taken with two different polarization settings.
The Output: For every single object, they now have a perfect "digital twin" that knows exactly how it reflects light, its true color (albedo), and its 3D shape (normals), completely separated from the lighting conditions.

4. Why This Matters: Teaching the AI to "See"

The researchers used this massive dataset to train AI models and tested them on three big tasks:

A. Intrinsic Decomposition (The "Magic Peel")

Goal: Take a photo of a shiny car in a parking lot and separate the car's paint color from the reflections of the sky and other cars.
Result: The AI trained on this real-world data became much better at "peeling" the image. It could tell the difference between a shiny spot caused by a light and a shiny spot caused by the material itself.

B. Relighting (The "Virtual Studio")

Goal: Take a photo of an object and change the lighting to look like it's in a sunset, a dark cave, or a neon city, without changing the object's shape or color.
Result: Previous AI models often made the shadows look fake or the colors shift weirdly. The new model, trained on real physics, creates lighting that looks photorealistic. The shadows fall correctly, and the shiny spots move naturally as the "virtual sun" moves.

C. 3D Reconstruction (The "Shape Shifter")

Goal: Build a 3D model of an object from just a few photos.
The Problem: Shiny objects confuse 3D scanners because the reflection moves when you move the camera, making the computer think the object is moving or changing shape.
The Fix: The AI uses the dataset to "predict" what the object looks like without the shiny reflections (the diffuse layer). It feeds this "clean" image to the 3D scanner.
Result: The 3D models built from these "clean" images are much smoother and more accurate, even if the original photos were full of confusing glare.

Summary

Think of this paper as the researchers building a giant, real-world physics textbook for computers.

Before, computers learned about light from "cartoons" (synthetic data). Now, they have a library of 1.2 million real photos where the "shiny" and "matte" parts of the world have been physically separated and measured. This allows AI to finally understand that a wet road isn't just a dark road, and a mirror ball isn't just a white ball—it's a masterclass in how light actually behaves in the real world.

1. Problem Statement

The core challenge in inverse rendering (intrinsic image decomposition) and forward rendering (relighting) is the scarcity of high-quality, real-world measured data.

Limitations of Synthetic Data: Current state-of-the-art models rely heavily on synthetic datasets (e.g., Objaverse, Hypersim). These suffer from simplified illumination models, limited material realism, and a lack of physically validated parameters (e.g., ignoring multi-scattering, polarization, and subsurface transport).
Generalization Gap: Models trained exclusively on synthetic data fail to generalize to uncontrolled real-world lighting and complex material interactions.
Limitations of Existing Real Datasets: Previous real-world datasets (e.g., Multi-Illumination Dataset) lack ground-truth intrinsic decompositions (albedo, normals, specular separation), making them unsuitable for supervised training of decomposition models.

2. Methodology

The authors propose a comprehensive solution involving the creation of a large-scale dataset and the training of neural rendering models based on it.

A. Data Acquisition System (Light Stage)

Hardware: An 8-camera, 346-light LED Light Stage equipped with cross/parallel linear polarizers.
Objects: 218 diverse everyday objects spanning organic produce, plastics, rubbers, metals, ceramics, glass, fabrics, wood, and paper.
Capture Protocol:
- Multiview: 8 synchronized viewpoints.
- Illumination: 346 "One-Light-at-a-Time" (OLAT) directions covering the hemisphere.
- Polarization: For every light/view combination, two images are captured: Cross-Polarized ( $I_\perp$ ) and Parallel-Polarized ( $I_\parallel$ ).
Resolution: 6144 × 3240 pixels, resulting in over 1.2 million high-resolution images.

B. Data Processing & Derivation

Using polarization physics (Malus's Law), the system physically separates reflections:

Diffuse/Specular Separation:
- Diffuse Radiance ( $I_d$ ) is derived from cross-polarized images: $I_d = 2 I_\perp$ .
- Specular Radiance ( $I_s$ ) is derived from the difference: $I_s = 2(I_\parallel - I_\perp)$ .
Material Attribute Extraction:
- Diffuse Albedo ( $\rho_d$ ) & Normals ( $n$ ): Solved by minimizing the error between observed diffuse radiance and Lambert's cosine law model.
- Specular Albedo ( $\rho_s$ ): Derived by integrating the specular reflection function over the captured lighting directions.
Lighting Augmentation: The OLAT data allows for the synthesis of arbitrary HDRI lighting by weighting and summing the OLAT captures based on environment map intensities.

C. Model Training

The authors fine-tuned state-of-the-art image-to-image translation models (specifically RGB2X and Diffusion Renderer) using Low-Rank Adaptation (LoRA) to prevent catastrophic forgetting. Two workflows were tested:

Physically-Based Rendering (PBR) Workflow: Predicts standard components (Diffuse Albedo, Specular Albedo, Surface Normals).
Polarization Workflow: Predicts polarization-separated components (Cross/Parallel images) to explicitly model reflection disentanglement, which aids in removing view-dependent specularities.

3. Key Contributions

First Large-Scale Real-World Polarized Dataset: The first dataset to provide dense, polarization-separated reflectance fields with ground-truth material attributes (albedo, normals, specular) for 200+ real objects.
Physical Grounding: Unlike synthetic data, this dataset captures real-world phenomena like polarization, multi-scattering, and complex BRDFs, enabling direct supervision for physically grounded neural rendering.
Versatile Acquisition Dimensions: The dataset spans five dimensions: Objects, Lighting (346 directions), Polarization, Multiview (8 views), and Material Attributes.
Demonstrated Generalization: Proved that training on real polarized data significantly improves model performance on real-world tasks compared to synthetic-only baselines.

4. Results and Evaluation

The models were evaluated on three tasks: Intrinsic Decomposition, Forward Relighting, and Sparse-View 3D Reconstruction.

Intrinsic Decomposition:
- Models trained on the new dataset achieved significantly lower MSE and higher PSNR/SSIM compared to baselines (RGB2X, DR-IR) trained on synthetic data.
- Key Finding: The polarization cues allowed for better disentanglement of diffuse and specular components under complex lighting, leading to smoother material separation.
Forward Relighting:
- PBR Workflow: Showed marked improvement in shading continuity and global color balance. On real objects, shadows aligned more faithfully with geometry, and highlight energy remained stable. PSNR improved from ~18.5 dB to 27.8 dB.
- Polarization Workflow: Demonstrated stronger lighting consistency and reduced noise in glossy regions, though it showed a slight domain gap when applied to synthetic data.
Sparse-View 3D Reconstruction:
- Using the inverse model to predict diffuse-only images (removing specularities) as input for 3D reconstruction algorithms (Dust3r, Mast3r) significantly reduced geometric errors.
- Result: The "Predicted Diffuse" input outperformed raw image inputs, achieving higher PSNR and SSIM, proving that polarization-based separation creates cleaner multiview correspondences for 3D reconstruction.

5. Significance

Bridging the Sim-to-Real Gap: This work establishes a new foundation for physically grounded material understanding, enabling models to generalize to real-world images where synthetic training fails.
Enabling New Applications: By providing a way to "virtualize" polarization (predicting polarized components from unpolarized inputs), the dataset enables advanced applications like robust 3D reconstruction in the wild and high-fidelity relighting without requiring specialized hardware.
Community Resource: The dataset (available at jingyang-carl.github.io/ICTPolarReal) fills a critical void in computer vision, offering the first large-scale resource for supervised training of inverse rendering models on real-world physics.

Limitations

Static Objects: The capture setup is limited to static objects; highly dynamic or moving objects cannot be captured.
Material Constraints: Highly transparent, strongly anisotropic, or complex subsurface scattering materials are difficult to measure accurately with the current setup.
Subsurface Parameters: The dataset does not explicitly capture subsurface scattering parameters.