💻 computer science

Do Image Editing Models Understand Lighting?

This paper introduces the 3D-anchored Light Probe (3DLP) benchmark, featuring a new high-fidelity HDR dataset of real-world lighting changes, to evaluate and reveal that while state-of-the-art image editing models demonstrate remarkable consistency with real-world physics, they still exhibit significant errors in dimly lit regions and lack the pixel-level precision required for accurate light transport analysis.

Original authors: Tim Küchler, Johann-Friedrich Feiden, Matthias Nießner, Carsten Rother

Published 2026-06-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tim Küchler, Johann-Friedrich Feiden, Matthias Nießner, Carsten Rother

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a magic paintbrush that can change anything in a photo just by listening to your voice. You can tell it, "Turn on the lamp," and it tries to do exactly that. But here's the big question: Does this magic brush actually understand how light works, or is it just guessing what a lit room looks like?

This paper, titled "Do Image Editing Models Understand Lighting?", is like a rigorous science test for these AI "magic brushes." The authors wanted to find out if these AI models truly grasp the physics of light (shadows, reflections, brightness) or if they are just mimicking patterns they've seen before.

Here is a simple breakdown of what they did and what they found:

1. The Problem: The "Fake It Till You Make It" Trap

Previous tests for these AI models were a bit like judging a magician by asking an audience, "Did that look real?" The audience might say "Yes!" because the lighting looks okay, but they might miss subtle physics errors, like a shadow falling in the wrong direction or a reflection that doesn't match the light source.

Also, some tests used computer-generated fake rooms (like a video game world). The authors argued that to really know if AI understands light, you need to test it in the messy, real world.

2. The Solution: The "Light Switch" Test (3DLP)

The authors built a new, super-strict test called 3DLP (3D-anchored Light Probe).

The Setup: They went into 1,000 different real-world rooms (offices, living rooms, etc.). In each room, there was a specific lamp with a round bulb.
The Action: They took two high-quality photos of the exact same scene: one with the lamp OFF and one with the lamp ON.
The Challenge: They gave the "OFF" photo to the AI and said, "Turn this light on." Then, they compared the AI's result to the real "ON" photo they took.

Think of it like a "Spot the Difference" game, but the AI has to generate the missing half of the picture from scratch.

3. The New Rules: Ignoring the "Photo Filter"

The authors realized that if an AI changes the light, it might also accidentally change the "white balance" (making the whole photo look warmer or cooler) or the exposure (making the whole image brighter or darker).

To be fair, they invented two new scoring rules:

The "Ratio" Trick: Instead of comparing the raw brightness, they compared the change caused by the light. It's like asking, "Did the AI add a shadow in the right spot?" rather than "Is the whole picture bright enough?"
The "Smoothness" Check: Real light fades away smoothly as it gets further from the source. The AI's shadows should fade smoothly too, not look jagged or noisy.

4. The Results: Who Passed the Test?

They tested six of the smartest image-editing AIs available.

The Winners: The commercial models Nano Banana Pro and Nano Banana 2 were the best. They were surprisingly good at understanding physics. They cast shadows in the right places, added realistic reflections on shiny surfaces, and didn't mess up the rest of the room.
The Runners-Up: The open-source model Qwen-Image-Edit did a great job too, beating the other open-source models.
The Strugglers: Some models (like Flux 2 Dev and GPT Image 1.5) made mistakes. They sometimes forgot to remove shadows when turning a light off, or they added reflections where they shouldn't exist.

Key Finding: The AI models are getting very good at the "big picture" of light. However, they still struggle in the "shadows" (literally). When a part of the room is far away from the light and stays dark, the AI is more likely to make mistakes there.

5. The "Human Eye" vs. The "AI Eye"

The authors also tested if Vision-Language Models (VLMs)—AI that can "see" and "read" images—could grade these results.

The Result: The VLMs were terrible at this specific job. They could tell if a picture looked "pretty" or "plausible" to a human, but they couldn't spot the tiny, pixel-level physics errors. It's like asking a poet to grade a math test; they can appreciate the story, but they can't check the equations.

6. The Bottom Line

The paper concludes that while these AI image editors are becoming incredibly realistic and are starting to understand the "physics" of light, they aren't perfect scientists yet. They are great at the "art" of lighting but still need to improve on the "math" of how light travels through a room.

The authors have released their dataset (the 1,000 real photos) and their scoring system to the public so other researchers can keep trying to teach these AIs how to be better physicists.

Technical Summary: Do Image Editing Models Understand Lighting?

Problem Statement
Recent advancements in instruction-based image editing models have achieved high visual fidelity, yet it remains unclear whether these systems possess an intrinsic understanding of real-world lighting physics or merely reproduce perceptual appearance priors. Existing benchmarks for evaluating these models typically rely on high-level plausibility assessments via Vision-Language Models (VLMs) or human judgment, or they utilize synthetically generated datasets. These approaches often fail to quantify pixel-level physical correctness, particularly regarding intricate light transport effects such as cast shadows, specular highlights, and interactions with complex materials. Furthermore, current evaluation metrics (e.g., PSNR, SSIM) are sensitive to global adjustments like white balance and exposure, which can obscure a model's actual performance in light transport simulation.

Methodology
To address these gaps, the authors introduce the 3D-anchored Light Probe (3DLP) benchmark, designed to evaluate light transport consistency using real-world, high-fidelity data.

Dataset Construction: The authors captured a new dataset of 1,000 high-fidelity HDR image pairs across diverse indoor environments. Each pair consists of a scene with a spherical light probe (a light bulb on a floor lamp) turned "off" and "on." The lamp base is visible to "anchor" the light source in 3D space, allowing models to infer precise position. The capture process involves 14-bit RAW imaging with 9-exposure brackets, averaged to reduce noise, and corrected for lens distortion and vignetting using EXIF data.
Annotation: The dataset includes manual annotations for challenging optical phenomena, including cast shadows, specular highlights, and interactions with metallic, mirrored, and transparent surfaces. Approximately 10.92% of pixels are labeled.
Evaluation Protocol: The task is binary: given an "off" image, the model must turn the light "on" (and vice versa). The input is an 8-bit sRGB image; the output is linearized for evaluation.
New Metrics: To overcome the limitations of traditional metrics, the authors propose two new scores invariant to global photographic adjustments (exposure, white balance, and light source intensity/color):
1. Standardised Intensity Error (SIE): Measures the accuracy of predicted light intensity changes. It maps ratio images ( $E = I_{on}/I_{off}$ ) to a standardised space using the Median and Median Absolute Deviation (MAD) to achieve invariance to affine transformations ( $C \odot E_R + K$ ).
2. Low-Frequency Error (LFE): Assesses the physical realism of light decay and surface interactions by evaluating the spatial gradients of the ratio images. It filters out high-frequency edges (texture/geometry) to focus on smooth transitions characteristic of inverse-square fall-off and Lambertian shading.

Key Contributions

3DLP Dataset: A high-fidelity dataset of 1K HDR image pairs capturing diverse real-world indoor scenes with a physically anchored light probe, including fine-grained annotations for shadows, highlights, and material interactions.
3DLP Benchmark & Metrics: A rigorous evaluation framework featuring SIE and LFE, which allow models to freely adjust global exposure and white balance while strictly penalizing errors in light transport physics.
Comprehensive Evaluation: A systematic evaluation of six state-of-the-art image editing models (including commercial and open-source variants) against real-world ground truth.

Results
The evaluation of six models (Nano Banana Pro, Nano Banana 2, Qwen-Image-Edit, Flux 2 Max, Flux 2 Dev, and GPT Image 1.5) reveals significant performance disparities:

Top Performers: The commercial model Nano Banana Pro achieved the best overall performance, followed closely by Nano Banana 2. Among open-source models, Qwen-Image-Edit significantly outperformed Flux 2 Dev.
Physics Alignment: Leading models demonstrate a remarkable consistency with real-world physics, particularly in casting new shadows and adding reflections. However, they still leave room for improvement.
Failure Modes:
- Intensity Dependence: Models perform worse in image regions receiving less light from the probe (distant areas), where errors accumulate.
- Material Complexity: Performance varies across materials; errors are most pronounced for mirrors and transparent surfaces, while specular highlights show slightly less variance.
- VLM Limitations: When tested as evaluators, VLMs (Gemini 3.1 Pro, GPT-5.5) failed to correlate with the physical metrics, often preferring AI-generated artifacts over real ground truth, indicating they are unsuitable for pixel-level light transport analysis.
Metric Comparison: Unlike PSNR, SSIM, and LPIPS, the proposed SIE and LFE metrics are invariant to white balance shifts and correctly penalize models that fail to execute the task (e.g., "Do Nothing" models).

Significance
The paper argues that while modern image editing models are rapidly becoming practical tools for professional visual media (e.g., inserting virtual products, relighting backgrounds), their underlying understanding of illumination remains ambiguous. The 3DLP benchmark provides a necessary, physically grounded standard to distinguish between models that genuinely simulate light transport and those that rely on perceptual priors. The authors posit that this distinction is vital for downstream applications such as intrinsic image decomposition, depth prediction, and light estimation, which implicitly rely on the base models' robustness to physical laws. By releasing the dataset and benchmark, the authors aim to encourage future research into improving the physical grounding of generative image editing.