Under One Sun: Multi-Object Generative Perception of Materials and Illumination

This paper introduces MultiGP, a generative inverse rendering method that leverages the shared illumination across multiple objects in a single image to stochastically disentangle and recover their individual reflectance, texture, and the common lighting conditions through a novel cascaded architecture and coordinated diffusion guidance.

The Gist

Imagine you walk into a room and see a shiny metal spoon, a matte ceramic mug, and a piece of rough wood sitting on a table. You take a single photo of them.

Now, try to answer this: What does the light in the room actually look like?

This is a classic puzzle for computers. The light hitting the spoon looks different than the light hitting the mug because the spoon reflects light like a mirror, while the mug scatters it like a foggy window. The wood absorbs some light and changes its color. In computer vision, this is called the "inverse rendering" problem: trying to figure out the cause (the light and materials) just by looking at the effect (the photo).

Usually, this is impossible to solve perfectly from just one picture. It's like trying to guess the flavor of a soup just by tasting a single spoonful; you might think it's salty, but maybe the salt is just on the surface, or maybe the spoon itself was salty.

Enter "MultiGP" (Multi-Object Generative Perception).

The researchers behind this paper came up with a clever solution: Don't look at just one object; look at the whole group.

Here is how they did it, explained with some everyday analogies:

1. The "Team Detective" Analogy

Imagine you are trying to figure out what the weather is outside, but you can only look through three different windows:

• Window A is covered in thick, blurry fog (like a matte object). It tells you it's bright, but not the direction of the sun.
• Window B is a perfect, clear mirror (like a shiny object). It shows a sharp reflection of the sun, but maybe the sun is hidden behind a tree in that specific reflection.
• Window C is a tinted glass (like a colored object). It shows the color of the sky but distorts the shape of the clouds.

If you only looked at Window A, you'd be guessing. If you only looked at Window B, you might miss details. But if you combine the clues from all three, you can reconstruct the entire weather scene perfectly.

MultiGP does exactly this. It looks at multiple objects in one photo. It knows that even though the spoon, mug, and wood look different, they are all being lit by the same sun. By comparing how the light bounces off the "foggy" mug versus the "mirror" spoon, the AI can mathematically strip away the confusion and figure out exactly what the light source looks like.

2. The "Magic Eraser" and the "Texture Scanner"

The AI has to do two difficult jobs at once:

1. Separate the "Paint" from the "Light": It needs to figure out the object's true color and texture (the paint) versus the shadows and highlights (the light).
2. Reconstruct the Light Source: It needs to build a 360-degree map of the room's lighting.

To do this, the paper describes a three-step process:

• Step 1: The Texture Stripper. First, the AI uses a "generative" model (think of it as a highly advanced magic eraser) to guess what the objects would look like if they had no lighting at all—just their raw colors and textures. It's like imagining the mug and spoon in a perfectly white, shadowless studio.
• Step 2: The Group Huddle (The "Cross-Talk"). This is the secret sauce. The AI takes the "shadowless" versions of all the objects and asks them to talk to each other.
  • The Analogy: Imagine a group of people trying to solve a puzzle where each person only has a few pieces. The person with the "shiny" object has the pieces showing the bright sun. The person with the "matte" object has the pieces showing the soft sky. They pass their pieces back and forth (this is called Axial Attention). By sharing what they see, they can build the complete picture of the sky.
• Step 3: The Reality Check. Finally, the AI takes its best guess of the light and the textures, puts them back together in a virtual 3D simulator, and renders a new image. It compares this new image to the original photo you gave it. If they don't match perfectly, it tweaks its guess and tries again. This ensures the final answer isn't just a pretty guess, but a physically accurate one.

3. Why is this a big deal?

Before this, computers usually tried to guess the light from a single object. It was like trying to guess the weather by looking at only one blurry window. The results were often blurry, wrong, or just "good enough."

MultiGP is the first method that says, "Hey, we have a whole team of objects here! Let's use their combined superpowers."

• It handles ambiguity: It admits that sometimes there isn't just one answer, so it generates many possible scenarios and finds the one that fits best.
• It's realistic: It doesn't just guess the light; it figures out the texture of the wood and the shine of the metal simultaneously.
• It works in the real world: They tested it on photos of real objects, and it worked surprisingly well, even with complex lighting.

The Bottom Line

Think of MultiGP as a master detective who realizes that to solve a crime (figuring out the lighting), you shouldn't just interview one witness (one object). You should interview the whole crowd. By listening to how the shiny witness, the matte witness, and the colored witness all describe the same event, the detective can reconstruct the truth with incredible accuracy.

This technology is a huge step forward for robots that need to understand the world, for virtual reality that needs to look real, and for any computer vision system that needs to "see" the light, not just the objects.

Technical Summary

1. Problem Statement

The paper addresses the fundamental challenge of inverse rendering: recovering the physical constituents of a scene (texture, reflectance, and illumination) from a single 2D image.

• The Ambiguity: Recovering these properties is an ill-posed problem. A single image results from the convolution of surface geometry, material reflectance (BRDF), and incident lighting. Different combinations of these factors can produce identical visual appearances (e.g., a dark object in bright light vs. a bright object in dim light).
• Limitations of Prior Work:
  • Deterministic methods often fail to capture the inherent ambiguity, producing a single "best guess" that may be physically incorrect.
  • Generative methods (diffusion models) have shown promise but typically focus on either shape/material or illumination, rarely handling all three simultaneously.
  • Single-object constraints: Previous generative inverse rendering methods (like DRM) struggle with textured objects or cannot disentangle texture from reflectance effectively when only one object is present.

Core Insight: While individual objects in a scene have unique textures and reflectances, they are all illuminated by the same global environment. MultiGP leverages this "multi-object consensus" to constrain the solution space, using complementary information from multiple objects to resolve ambiguities that exist for any single object.

2. Methodology: Multi-Object Generative Perception (MultiGP)

MultiGP is a stochastic generative inverse rendering framework that samples the joint posterior distribution of texture ($T$), reflectance ($R$), and illumination ($L$) given a single image ($I$) and known object shapes ($S$).

The architecture employs a cascaded end-to-end factorization (Eq. 4), separating the problem into two stages:

A. Texture Extraction Prior ($q_\phi$)

• Goal: Isolate the spatially varying diffuse texture ($T$) from the input image.
• Mechanism: A Latent Diffusion Model conditioned on the concatenated latents of the observed appearance and object shapes.
• Output: Produces a texture-free appearance and an initial estimate of the texture, effectively removing high-frequency surface details from the radiometric analysis in the next stage.

B. Multi-Object Diffusion Reflectance Maps ($q_\theta$)

• Goal: Jointly estimate the shared illumination ($L$) and per-object reflectances ($R$) using the texture-free inputs.
• Representation: Objects are mapped to Reflectance Maps (Gaussian spheres indexed by surface normal), which represent the raw radiance independent of specific viewing angles.
• Key Technical Components:
  1. Coordinated Guidance (Coordinate Scheduling):
     • To ensure all $M$ objects converge to a single consistent illumination estimate, the diffusion process schedules the reflectance of each object to evolve linearly from its estimated material state toward a known "mirror" reflectance state over $K$ steps.
     • This forces the diverse material responses to stochastically converge on the same environment map, unlike single-object methods that might drift.
  2. Multi-Object Axial Attention:
     • Different materials act as frequency filters (e.g., matte surfaces filter out high-frequency lighting details; shiny surfaces preserve them).
     • This mechanism facilitates "cross-talk" between objects at corresponding surface normal directions. It allows an object with missing frequency details (due to its material) to borrow information from other objects in the scene, creating a complete, high-fidelity illumination estimate.
  3. Texture Extraction ControlNet:
     • A refinement step that ensures physical consistency. It renders the estimated factors ($T, R, L$) using a differentiable renderer (Mitsuba 3) and computes the residual against the original image.
     • This residual is fed back into the diffusion process via a ControlNet, steering the sampling toward solutions that strictly obey the physics of image formation without sacrificing generative diversity.

3. Key Contributions

1. Multi-Object Consensus Framework: The first generative inverse rendering method to simultaneously sample texture, reflectance, and illumination by leveraging the shared illumination constraint across multiple objects.
2. Novel Architecture:
   • Cascaded Factorization: Separates texture (image domain) from reflectance/illumination (angular domain).
   • Coordinated Guidance: A scheduling mechanism ensuring multi-object convergence to a single environment map.
   • Axial Attention: A mechanism to fuse complementary spatio-spectral information across different materials.
   • ControlNet Refinement: Enforces physical consistency via renderer residuals.
3. Ambiguity-Aware Evaluation Metric:
   • Recognizing that inverse rendering is probabilistic, the authors propose a new metric based on Spherical Harmonics (SH) and Mahalanobis distance.
   • Instead of measuring the distance to a single ground truth, they evaluate how well the distribution of generated samples encompasses the ground truth within a PCA space. This better captures the "correctness" of stochastic sampling.

4. Experimental Results

The method was evaluated on synthetic datasets (Adobe 3D Assets, nLMVS-Synth) and real-world datasets (Stanford-ORB, nLMVS-Real, and a newly captured MultiGP dataset).

• Quantitative Performance:
  • Illumination: MultiGP achieved state-of-the-art (SOTA) results in logRMSE, PSNR, and SSIM, significantly outperforming single-object baselines (DRM) and other generative methods (DiffusionLight, DPI).
  • Texture: Outperformed existing texture estimation methods (RGB↔X, DiffusionRenderer) with lower RMSE and higher SSIM.
  • Ablation Studies: Removing Coordinated Scheduling or Axial Attention resulted in measurable drops in accuracy, confirming their necessity for handling ambiguity.
• Qualitative Analysis:
  • Frequency Integration: Visualizations of Spherical Harmonic coefficients show that MultiGP successfully integrates complementary frequency bands from different objects (e.g., using a shiny object to recover high-frequency lighting details lost on a matte object).
  • Real-World Robustness: The method successfully recovered complex lighting structures in real-world scenes with varying geometries and materials, even where ground truth was unavailable.
• Distribution Quality: The proposed ambiguity-aware metric showed that the MultiGP distribution has a significantly higher likelihood of containing the ground truth compared to single-object estimates.

5. Significance and Conclusion

• Paradigm Shift: MultiGP moves inverse rendering from deterministic "best guess" estimation to probabilistic, ambiguity-aware sampling. It acknowledges that multiple physical explanations can exist and provides a distribution of valid solutions.
• Physical Consistency: By leveraging the physical constraint that "one sun lights all objects," the method resolves the "chicken-and-egg" problem of disentangling material and light more effectively than single-object approaches.
• Future Directions: The authors note current limitations, specifically the requirement for known 3D geometry and the assumption of distant illumination. Future work aims to relax these constraints to handle near-field lighting and joint shape estimation.

In summary, MultiGP represents a significant advancement in computer vision by demonstrating that the collective information of multiple objects in a single image can be harnessed via generative models to achieve robust, high-fidelity, and physically consistent scene understanding.