🤖 AI

CLONE: A 3DGS-Based Closed-Loop Differentiable Optimization Framework for Single-Image Normal Estimation

The paper proposes CLONE, a closed-loop differentiable optimization framework that leverages 3D Gaussian Splatting, a learnable illumination model, and a diffusion-inspired refinement network to achieve high-quality single-image normal estimation without ground-truth supervision by enforcing image-geometry-image consistency.

Original authors: Yanxing Liang, Yinghui Wang, Wei Li, Tao Yan, Jiaxing Shen

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yanxing Liang, Yinghui Wang, Wei Li, Tao Yan, Jiaxing Shen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking at a flat, two-dimensional photograph of a 3D object, like a ceramic bowl or a toy car. Your brain instantly knows how the surface curves, where the bumps are, and how the light hits it. But for a computer, this is a massive puzzle. The photo has lost all the depth information; it's like trying to guess the shape of a sculpture just by looking at its shadow on a wall. This is called an "ill-posed problem" because there are many possible shapes that could create that same flat image.

This paper introduces a new system called CLONE to solve this puzzle. Instead of just guessing the shape based on patterns it has memorized (like a student memorizing answers), CLONE acts like a smart architect who builds a model, checks the light, and fixes the errors.

Here is how CLONE works, broken down into simple steps:

1. The "Clay" Model (3D Gaussian Splatting)

Most computer vision methods try to guess the surface directly from the pixels. CLONE does something different: it starts by building a physical 3D model of the object using thousands of tiny, invisible "blobs" of light (called 3D Gaussians).

The Analogy: Imagine you are trying to recreate a statue. Instead of painting a flat picture of it, you start with a cloud of clay balls. The computer arranges these balls to roughly match the shape of the object in the photo. Because these balls have mathematical properties, the computer can instantly calculate which way the surface is facing (the "normal") just by looking at how the balls are stretched.

2. The "Flashlight Test" (Differentiable Optimization)

Here is the magic trick. Once the computer has built this 3D clay model, it doesn't just stop there. It shines a virtual flashlight on its own model and takes a new picture of it.

The Loop: It compares this new "fake" picture with the original photo you gave it.
- If the fake picture looks different (e.g., the shadow is wrong, or the highlight is in the wrong spot), the computer knows its 3D model is slightly off.
- It then automatically adjusts the 3D clay balls to make the fake picture match the real one better.
Why this is special: Most methods need a teacher to say, "You are wrong, here is the correct answer." CLONE doesn't need a teacher. It learns by trying to make its own 3D model look exactly like the 2D photo. If the model is wrong, the "shadow" in the photo will be wrong, and the computer fixes the model to fix the shadow. This creates a closed loop of self-correction.

3. The "Detail Polisher" (Diffusion Refinement)

The "clay ball" model is great for getting the big shape right, but it tends to be a bit smooth and blurry, like a low-resolution sculpture. It misses tiny details like the veins on a leaf or the keys on a keyboard.

The Analogy: Think of the clay model as a rough draft. CLONE then uses a "detail polisher" (a one-step diffusion network) to sharpen the edges.
The Gating Mechanism: This polisher is smart. It has a "gate" that decides when to trust the clay model and when to add new details.
- If the clay model is already perfect (like a smooth sphere), the gate says, "Leave it alone."
- If the clay model is too smooth (like a flat keyboard), the gate says, "Add the sharp edges here."
- This ensures the computer doesn't invent fake details where they don't belong, but it also doesn't miss real details.

4. The Result: No Teacher Needed

The biggest breakthrough of CLONE is that it doesn't need ground-truth labels.

Old Way: To train a computer to see 3D shapes, you usually need thousands of photos where humans have manually drawn the exact 3D angles for every single pixel. This is expensive and slow.
CLONE Way: It only needs pairs of photos and 3D models (which are easy to find online). It uses the photo to check if its 3D model is correct. It learns the "rules of light and shape" on its own.

Summary

Think of CLONE as a self-correcting sculptor:

It builds a rough 3D statue from a photo.
It shines a light on its own statue and compares the shadow to the original photo.
If the shadow doesn't match, it reshapes the statue.
It uses a special tool to sharpen the tiny details without messing up the big shape.
It does all this without ever being told the "right answer" by a human.

The paper claims that this method is more accurate than previous state-of-the-art methods, even beating systems that were trained with expensive human labels. It works well on smooth objects, complex textures, and thin structures, proving that this "closed-loop" way of thinking about 3D shapes is a powerful new tool for computers.

Technical Summary: CLONE – A 3DGS-Based Closed-Loop Differentiable Optimization Framework

Problem Statement
Single-image surface normal estimation is an inherently ill-posed geometric problem, as inferring 3D structure from a single 2D observation admits multiple plausible solutions. Existing approaches suffer from fundamental limitations:

Discriminative methods rely heavily on expensive, dense ground-truth normal labels and lack intrinsic geometric modeling, leading to poor generalization under distribution shifts.
Generative methods (e.g., diffusion models) leverage strong distribution priors but often lack explicit geometric variables and stable, differentiable pathways to propagate image-space errors back to 3D geometry.
Explicit geometric methods (e.g., 3D Gaussian Splatting variants) provide differentiable rendering but often fail to formulate normal estimation as a closed-loop optimization problem in single-view settings, lacking sufficient constraints to uniquely determine geometry without multi-view supervision.

The core challenge is the absence of a unified, differentiable optimization mechanism that simultaneously enforces solution ambiguity constraints and geometric consistency without requiring ground-truth normal labels.

Methodology
The authors propose CLONE (Closed-Loop differentiable Optimization framework for single-image Normal Estimation), which reformulates normal estimation from a direct regression task to a geometric optimization problem driven by reprojection consistency. The framework operates without pixel-level normal supervision, utilizing only image-3D model pairs.

3DGS-Based Explicit Representation & Analytical Normals:
The scene is parameterized using 3D Gaussian Splatting (3DGS). Unlike standard 3DGS used for color reconstruction, CLONE explicitly derives surface normals analytically via eigen-decomposition of the Gaussian covariance matrix ( $\Sigma = U\Lambda U^\top$ ). The eigenvector corresponding to the smallest eigenvalue defines the local surface normal. This provides a continuous, differentiable pathway from 3D geometric parameters to surface orientation.
Differentiable Illumination & Reprojection:
To bridge 3D geometry and 2D observations without normal labels, the authors introduce a differentiable illumination model. They formulate a continuous mapping between surface normals and image radiance using a learnable light modulation kernel (an oscillatory kernel with a Gaussian envelope). This allows the model to simulate how local geometric variations affect pixel intensity. A reprojection loss ( $L_{photo}$ ) is computed between the rendered image (generated by blending Gaussian contributions) and the input image, enabling gradients to flow directly from image-space errors back to 3D geometric parameters.
Conditional Diffusion-Inspired Refinement:
To address the over-smoothing inherent in Gaussian primitives, a one-step deterministic diffusion-inspired network is employed. Instead of a multi-step generative process that truncates gradients, this module acts as a noise-conditioned residual refinement operator. It takes the initial 3DGS normals and injects noise at a fixed step, then uses a U-Net to predict a refined normal map. This preserves the structural inductive bias of diffusion models while maintaining end-to-end differentiability.
Cross-Domain Gating Fusion:
A gating mechanism coordinates the global geometric consistency of the 3DGS branch and the high-frequency detail recovery of the diffusion branch.
- Feature Level: Geometric features (from 3DGS) and texture features (from the diffusion encoder) are aligned and fused via an adaptive attention-guided gating mechanism.
- Output Level: A spatially adaptive gating weight ( $g$ ) fuses the initial geometric normal ( $n_{3dgs}$ ) and the refined normal ( $n_{diff}$ ). The final fused normal is $n_{fuse} = \text{stopgrad}(g) \odot n_{3dgs} + (1 - \text{stopgrad}(g)) \odot n_{diff}$ . This ensures that photometric loss supervises both branches without causing gradient conflicts.
Unified Optimization Objective:
The system is optimized end-to-end using a composite loss function:
- Photometric Loss ( $L_{photo}$ ): Ensures image reconstruction consistency.
- Scale Regularization ( $L_{scale}$ ): Constrains Gaussian eigenvalues to prevent degenerate shapes.
- Normal Consistency ( $L_{normal}$ ): Encourages the refined normal to stay close to the geometric normal in regions where the gating weight favors the geometric prior.
- Directional Regularization ( $L_{dir}$ ): Aligns learnable principal directions with the geometric normals.

Key Contributions

Paradigm Shift: The paper reformulates single-image normal estimation as a closed-loop differentiable optimization problem, eliminating the need for ground-truth normal labels by using reprojection errors as the primary supervision signal.
Analytical Normal Modeling: It proposes a method to derive normals analytically from 3D Gaussian covariance, optimizing in a continuous geometric space rather than relying solely on 2D feature regression.
Differentiable Refinement: A one-step deterministic diffusion-inspired network is designed to enhance high-frequency details without the gradient truncation issues associated with multi-step diffusion sampling.
Unified Pipeline: The integration of 3DGS parameterization, differentiable rendering, and conditional diffusion refinement creates a stable gradient flow, effectively alleviating the ill-posed nature of the problem.

Experimental Results
The method was evaluated on three benchmarks: Google Scanned Objects (GSO), Omniobject3D, and Wonder3D, using only RGB-3D model pairs for training (no normal labels).

Quantitative Performance: CLONE achieves state-of-the-art performance, outperforming both supervised discriminative methods and generative 3D reconstruction baselines. It achieves a Mean Angle Error (MAE) of 13.2°, surpassing the second-best method (Wonder3D++) by 0.3°. It also leads in Median Angle Error (MedAE) and accuracy at strict thresholds (e.g., 63.2% accuracy at <11.25°).
Qualitative Analysis: Visual comparisons show CLONE effectively recovers fine geometric details (e.g., keyboard keys, leaf veins, feather patterns) and maintains consistent normals on thin structures and complex junctions, where baselines suffer from over-smoothing, artifacts, or broken silhouettes.
Robustness: The method demonstrates superior robustness to input noise, lighting variations, and viewpoint changes compared to baselines, attributed to its explicit 3D geometric constraints.
Ablation Studies: Removing any core component (3DGS initialization, cross-domain alignment, gating fusion, or diffusion refinement) leads to significant performance degradation, confirming the necessity of the closed-loop design.

Significance and Claims
The authors claim that CLONE offers a new paradigm for single-image normal estimation that is independent of expensive normal annotations. By leveraging large-scale image-3D model pairs and a closed-loop differentiable optimization framework, the method transforms the problem from a data-hungry mapping task into a geometry-centric optimization task. The paper asserts that this approach successfully addresses the ill-posedness of the problem by directly converting image-level errors into optimization signals for 3D geometry, achieving performance comparable to or exceeding fully supervised baselines without requiring any normal labels. This validates the potential of explicit 3D representation combined with differentiable rendering for annotation-efficient 3D geometry learning.