⚡ electrical engineering

The frame-level leakage trap: rethinking evaluation protocols for intrinsic image decomposition, with source-separable uncertainty as a case study

This paper exposes a significant evaluation leakage in intrinsic image decomposition caused by frame-level dataset splits, advocates for scene-level splits as the new standard, and introduces a physics-informed model with source-separable uncertainty that achieves competitive performance while effectively identifying and filtering high-error pixels.

Original authors: Jihwan Woo

Published 2026-05-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jihwan Woo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to separate a photograph into two distinct layers: the color of the object itself (like the red of a stop sign) and the lighting hitting it (like the shadow cast by a tree). This is called "intrinsic image decomposition." It's a bit like trying to guess how much of a room's brightness comes from the paint on the walls versus the lamp in the corner.

This paper tackles two main problems with how researchers currently test their computer programs for this task.

1. The "Cheat Sheet" Problem (Data Leakage)

For years, researchers have been testing their AI models by splitting a movie dataset (called MPI Sintel) into "training" and "testing" groups. However, they often made a mistake: they split the movie frame-by-frame.

The Analogy: Imagine you are studying for a history exam.

The Wrong Way (Frame-level split): You study Chapter 1, and the test asks you questions about Chapter 1, but just a few sentences later in the book. Since the story hasn't changed much, you get a perfect score. You aren't actually learning history; you're just memorizing the immediate next page.
The Right Way (Scene-level split): You study Chapter 1, but the test asks you about Chapter 10, a completely different scene with different characters and lighting.

The Discovery: The authors found that the "Wrong Way" was inflating scores by a huge margin (1.6 to 2.0 decibels, and even more with longer training). It was like giving students a cheat sheet. They proved that when you use the "Right Way" (testing on entirely new scenes), the scores drop significantly. They are urging the whole community to stop using the cheat sheet and start using the harder, fairer test.

2. The "Confidence Meter" (Uncertainty)

Most AI models just give you an answer: "This pixel is red." They don't tell you if they are sure. But in tricky situations—like a shiny car hood (specular highlight) or a textured wall—the AI might be guessing.

The authors built a new model that doesn't just guess; it also has a three-part confidence meter. Instead of just saying "I'm 50% sure," it breaks down why it's unsure:

Texture Confusion: "I'm unsure because the pattern looks like a shadow."
Lighting Confusion: "I'm unsure because the light is coming from a weird angle."
Reflection Confusion: "I'm unsure because this looks like a shiny reflection, not the object's true color."

The Proof:

Specialization: They showed that the "Reflection Confusion" meter actually lights up exactly when there are shiny spots in the image. It's not just a generic "I'm confused" light; it's a specific tool for specific problems.
Usefulness: They tested if this confidence meter could actually help. They told the computer to ignore the 75% of the image where it was most confused. The result? The remaining image was 77% more accurate than if they had just randomly ignored pixels. This proves the confidence meter is actually useful, even if it's not perfect.

3. The "More is Less" Lesson

The authors tried building a super-complex version of their model, adding five extra fancy features (like frequency decomposition and contrastive learning). They thought, "More tools must mean better results."

The Result: The fancy, complex model actually performed worse than the simpler one.

The Lesson: Just because you can add more features doesn't mean you should. Sometimes, a simple, honest approach works better than a complicated one that tries to do too much. They tested every single extra feature individually, and none of them helped on their own.

Summary

The paper is a call for honesty in AI research:

Stop cheating: Don't test your AI on data that is too similar to what it studied. Use the "Scene-level" split to get real results.
Know your limits: It's better to have a model that tells you where it is confused (and why) than a model that confidently gives the wrong answer.
Keep it simple: Adding more complex parts to a model doesn't always make it better; sometimes it makes it worse.

The authors provide a new, fairer set of "reference scores" for the community and a working model that can tell you not just what the image is, but where it might be wrong.

Technical Summary: The Frame-Level Leakage Trap and Source-Separable Uncertainty in Intrinsic Image Decomposition

1. Problem Statement

Intrinsic image decomposition aims to factorize an observed image $I$ into reflectance (albedo) $R$ and shading $S$ (under the Lambertian assumption $I = R \circ S$ ). Despite progress in deep learning, current methods face two critical limitations:

Evaluation Protocol Inconsistency: Prior works on the MPI Sintel dataset frequently utilize frame-level train/test splits. This allows spatially similar frames from the same scene to appear in both training and testing sets, creating data leakage that artificially inflates performance metrics.
Lack of Uncertainty Quantification (UQ): Existing methods produce point estimates without quantifying confidence. This is problematic because decomposition is inherently ill-posed; ambiguity varies spatially (e.g., specular highlights, cast shadows, and textured surfaces). In safety-critical applications like autonomous driving, knowing where a decomposition is unreliable is as vital as the decomposition itself.

2. Methodology

2.1 Corrected Evaluation Protocol

The authors advocate for scene-level splits as the community standard. In this protocol, entire scenes are held out for testing, ensuring no temporal or spatial overlap between training and test sets. This eliminates the "frame-level leakage" where models interpolate between adjacent frames of the same scene.

2.2 Physics-Informed Decomposition Model

The paper proposes a three-component image formation model to handle non-Lambertian effects:
$I = R \circ S + N$
Where $N$ is a non-negative residual capturing specular highlights, subsurface scattering, and other non-Lambertian effects. This allows the reflectance and shading heads to focus on their respective Lambertian components while $N$ handles deviations.

2.3 Network Architecture

The backbone is a U-Net encoder-decoder. It branches into four heads:

Reflectance Head ( $\hat{R}$ ): Predicts albedo.
Shading Head ( $\hat{S}$ ): Predicts illumination.
Non-Lambertian Head ( $\hat{N}$ ): Predicts the residual $N$ using ReLU activation.
Source-Separable Uncertainty Head: Outputs three log-variance channels ( $\log \sigma^2_{tex}, \log \sigma^2_{light}, \log \sigma^2_{nl}$ $lo g σ_{t e x}^{2}, lo g σ_{l i g h t}^{2}, lo g σ_{n l}^{2}$ ) corresponding to texture, lighting, and non-Lambertian sources.
- Design Choice: The uncertainty head receives a skip connection from the raw input image $I$ alongside the bottleneck features. This preserves high-frequency pixel-level cues (e.g., saturation, gradients) lost during encoding, which are critical for detecting ambiguity.

2.4 Training Strategy

To prevent "variance explosion" (where the network minimizes loss by predicting infinite uncertainty everywhere), a two-phase training strategy is employed:

Phase 1 (Warmup): Train only reconstruction heads ( $R, S, N$ ) for 55 epochs. The uncertainty head is frozen.
Phase 2 (NLL): Gradually introduce the heteroscedastic Negative Log-Likelihood (NLL) loss for 25 epochs, jointly training all parameters.

3. Key Contributions

Quantification of Frame-Level Leakage: The paper provides the first quantitative demonstration that frame-level splits on MPI Sintel inflate test R PSNR by 1.6–2.0 dB (statistically significant, $p < 0.01$ ) compared to scene-level splits across three architectures (DirectCNN, UNet, and the proposed method). Under extended training, this gap exceeds 10 dB.
Source-Separable Uncertainty: A novel three-way uncertainty head that decomposes uncertainty into texture, lighting, and non-Lambertian sources. The authors empirically verify that these channels do not collapse to a shared scalar; specifically, the non-Lambertian channel ( $\sigma_{nl}$ ) shows a cross-correlation of $r=0.67$ with non-Lambertian residual error, more than 4 $\times$ stronger than the texture channel's correlation with the same error.
Downstream Utility of Uncertainty: The paper demonstrates that uncertainty rankings are practically informative. Filtering out the 75% highest-uncertainty pixels reduces reconstruction MSE by 77% on the retained pixels, whereas random filtering yields no improvement.
Negative Results and Ablation: The authors report that a more elaborate variant combining frequency decomposition, evidential learning, contrastive loss, and test-time adaptation (TTA) performs worse than the simpler baseline. Leave-one-out ablation confirms that no individual component of this complex variant improves the baseline, suggesting that feature accumulation without disciplined evaluation can be detrimental.

4. Experimental Results

4.1 Quantitative Performance (Scene-Level Split)

Under the strict scene-level protocol:

Proposed Method: Achieves 15.98 ± 0.41 dB R PSNR and 16.09 ± 0.18 dB S PSNR.
Comparison: This performance is statistically indistinguishable from a UNet+Physics baseline (15.97/16.00 dB) but significantly lower than a 5-member Deep Ensemble (16.78/16.51 dB).
Cost Efficiency: The proposed method achieves near-ensemble performance at one-fifth the parameter and training cost.
Significance: None of the incremental architectural changes (Physics head, skip connections) produced statistically significant PSNR gains ( $p > 0.05$ ) in paired t-tests across seeds, reinforcing the paper's thesis that marginal architectural tweaks yield marginal gains under honest evaluation.

4.2 Uncertainty Analysis

Calibration: The global uncertainty-error correlation is modest ( $r = 0.131$ ), comparable to or slightly better than Deep Ensembles ( $r = 0.107$ ) and consistent with benchmarks in optical flow and depth estimation.
Specialization: The non-Lambertian channel ( $\sigma_{nl}$ ) strongly correlates with non-Lambertian errors ( $r=0.67$ ), while the texture channel ( $\sigma_{tex}$ ) does not. This source-specific specialization is unique compared to scalar uncertainty estimators.
Generalization: The source-separable behavior persists on out-of-distribution real photographs (no fine-tuning), with $\sigma_{nl}$ showing positive correlation with predicted non-Lambertian residuals.

4.3 Negative Findings

Complex Variants: A variant combining five advanced mechanisms (frequency routing, cross-task supervision, evidential learning, contrastive loss, TTA) resulted in a 3.4 dB drop in S PSNR and negative uncertainty-error correlation.
Ablation: Leave-one-out analysis showed that no single component of the complex variant improved the baseline, indicating that the failure was not due to one "bad" component but the incompatibility of the combined mechanisms.

5. Significance and Claims

The paper argues that the intrinsic image decomposition community has been misled by overly permissive evaluation protocols. The primary significance lies in:

Protocol Correction: Establishing scene-level splits as the necessary standard to prevent data leakage, providing reference numbers for six models under this protocol.
Interpretability over Raw Metrics: Demonstrating that while the proposed method does not drastically outperform simpler baselines in PSNR, its source-separable uncertainty provides unique, actionable information (e.g., distinguishing specular highlights from texture) that scalar ensembles cannot offer.
Disciplined Evaluation: The paper cautions against "feature accumulation" (adding complex modules) without rigorous ablation. It suggests that for ill-posed inverse problems, disciplined architectural restraint often yields more robust results than complex, unverified combinations of mechanisms.

The authors conclude that uncertainty quantification is a natural, underexplored complement to intrinsic decomposition, offering tools for downstream tasks like relighting and material editing where knowing the limits of the decomposition is critical.