ReLi3D: Relightable Multi-view 3D Reconstruction with Disentangled Illumination
ReLi3D introduces a unified, end-to-end pipeline that leverages multi-view constraints and a novel two-path prediction strategy to simultaneously reconstruct complete 3D geometry, spatially-varying materials, and environment illumination from sparse images in under one second, enabling the near-instantaneous generation of relightable 3D assets.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you have a few photos of a toy car sitting on your desk. You want to turn those flat photos into a 3D model you can spin around on your computer. But here's the catch: you don't just want a gray plastic shell; you want the car to look like it has real metal paint, rubber tires, and glass windows. Plus, you want to be able to move the "sun" in your computer room and watch the shadows and reflections change realistically.
For a long time, doing this was like trying to solve a giant puzzle where half the pieces were missing. You had to build the shape, guess the materials, and figure out the lighting as three separate, slow, and complicated tasks.
Enter ReLi3D. Think of it as a "magic 3D scanner" that does all three jobs at once, in less than a second.
Here is how it works, broken down with some everyday analogies:
1. The Problem: The "Shadow vs. Paint" Confusion
Imagine looking at a dark spot on a wall. Is that spot dark because the wall is painted black, or because a lamp is off and casting a shadow?
- Old methods (Single-view AI) look at just one photo and guess. They often get confused. They might think a shiny metal car is actually a dull, dark plastic car because the lighting in the photo was dim. This is called an "ill-posed problem"—there are too many possible answers, and the AI picks the wrong one.
- ReLi3D's Solution: It looks at multiple photos taken from different angles. If you see the same spot on the car from the front, the side, and the top, and it looks shiny in all of them, the AI knows, "Aha! This is shiny metal, not a shadow." By combining all the views, it solves the puzzle instantly.
2. The Secret Sauce: The "Two-Path" Brain
ReLi3D has a special brain architecture that splits the work into two parallel paths, like a chef with two hands:
- Hand 1 (The Sculptor): This path builds the shape (the mesh) and figures out what the object is made of (the paint, the roughness, the metal). It creates a 3D model with "spatially varying" materials, meaning the hood can be shiny metal while the tires are matte rubber.
- Hand 2 (The Lighting Detective): This path looks at the background and the reflections on the object to figure out where the light is coming from. It reconstructs the entire "sky" or room lighting (called an HDR environment) that was in the original photo.
These two hands talk to each other constantly. If the Sculptor thinks a part is shiny, the Lighting Detective checks if the reflection matches the light source. If they disagree, they adjust until everything makes physical sense.
3. The Training: Learning from "Fake" and "Real"
To teach this AI, the researchers didn't just show it real photos. They used a "mixed diet":
- Synthetic Data (The Classroom): They used computer-generated 3D worlds where they knew the exact truth (e.g., "This is 100% red paint, lit by a blue sun"). This taught the AI the rules of physics.
- Real Data (The Real World): They then fed it messy, real-world photos from the internet. Because the AI already learned the rules of physics in the "classroom," it could now figure out the messy real-world photos without getting confused.
4. The Result: Instant, Relightable Magic
The coolest part? Once ReLi3D builds the model, it's relightable.
- Before: If you wanted to see your 3D car in a sunset, you'd have to re-render the whole thing, which takes minutes or hours.
- With ReLi3D: Because it separated the "object" from the "light," you can instantly swap the lighting. You can turn a sunny day into a rainy night, and the car's metal will still reflect the new moonlight correctly, and the shadows will shift instantly.
Summary
ReLi3D is like a super-fast, super-smart 3D artist that looks at a few photos, figures out exactly what the object is made of and where the light was, and hands you a fully realistic 3D model in 0.3 seconds. It solves the confusion between "dark paint" and "shadows" by looking at the object from all sides at once, allowing you to create 3D assets that look real and can be lit up in any way you want, instantly.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.