🤖 AI

Differentiable Inverse Graphics for Zero-shot Scene Reconstruction and Robot Grasping

This paper introduces a differentiable neuro-graphics model that enables zero-shot, physically consistent scene reconstruction and robot grasping from a single RGBD image by combining neural foundation models with physics-based differentiable rendering, eliminating the need for extensive training data or test-time samples.

Original authors: Octavio Arriaga, Proneet Sharma, Jichen Guo, Marc Otto, Siddhant Kadwe, Rebecca Adam

Published 2026-02-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Octavio Arriaga, Proneet Sharma, Jichen Guo, Marc Otto, Siddhant Kadwe, Rebecca Adam

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a robot walking into a room you've never seen before. There are objects on the table—a strange mug, a weirdly shaped fruit, a tool you don't recognize. In the past, for a robot to pick these up, it would need to have "studied" millions of pictures of similar items beforehand, or it would need to take hundreds of photos of the new object from every angle just to figure out what it is. It's like trying to guess the shape of a mystery box only after looking at a million other boxes.

This paper introduces a new way for robots to learn instantly, without all that heavy studying. The authors call their method Differentiable Neuro-Graphics. Here is how it works, using simple analogies:

The Problem: The "Black Box" vs. The "Physical Model"

Most modern AI is like a black box. You feed it data, and it guesses the answer. To make a good guess, it needs a massive library of examples (training data). If you show it a new object it hasn't seen, it often fails or needs to take many photos of it first to "learn" it on the spot.

This paper proposes a different approach: a physical model. Instead of just guessing based on patterns, the robot tries to understand the physics of the scene. It asks: "If I assume this object is a sphere made of metal under this light, does the picture I 'see' in my head match the picture my camera sees?"

The Solution: A Three-Step Detective Story

The system acts like a detective solving a crime scene with only one clue (a single photo). It follows a specific, step-by-step process to reconstruct the 3D world:

1. The "Sketch" Phase (Segmentation)
First, the robot looks at the photo and asks, "Where are the objects?" It uses a pre-trained "foundation model" (a very smart AI that knows what objects look like generally) to draw outlines around the items. It's like a child tracing the silhouette of a toy on a piece of paper.

2. The "Ball" Phase (Ellipsoid Estimation)
Next, the robot makes a rough guess about the 3D shape. It doesn't try to build a detailed sculpture yet. Instead, it imagines each object is a simple, stretchy balloon (an ellipsoid). It uses the depth information from the camera to figure out how big and where these balloons are.

The Trick: The authors found that starting with these "balloons" is crucial. If they tried to guess the final shape immediately, the robot would get confused and get stuck in a "local minimum" (a wrong answer that looks good but isn't). The balloon acts as a sturdy foundation.

3. The "Sculptor" Phase (Differentiable Rendering)
This is the magic part. The robot has a virtual camera inside its brain. It takes its current "balloon" guess and renders a fake image. Then, it compares this fake image to the real photo.

The Feedback Loop: If the fake image is too dark, the robot adjusts the "lighting" in its brain. If the fake object is too round, it stretches the "balloon" into a cube. It does this mathematically, over and over, refining the shape, the material (is it shiny or matte?), and the position until the fake image matches the real one perfectly.
The Mesh: Once the balloon is the right size and position, the robot swaps the balloon for a detailed 3D mesh (a wireframe model) and polishes it until it fits the object's curves perfectly.

Why This Matters for Robots

The paper claims this method allows a robot to:

See in "Zero-Shot": It can handle completely new objects it has never seen before, without needing a database of 3D models or extra photos.
Be "Explainable": Because the robot is building a physical model (light, material, shape), we can see why it thinks an object is there. It's not a magic guess; it's a calculated reconstruction.
Grasp Objects: The authors tested this by having a robot arm pick up real, unseen objects (like a baseball, a soup can, or a wine glass). Because the robot had built an accurate 3D model of the object in its "mind," it could calculate exactly where to grab it.
- The Result: In their tests, the robot successfully grabbed the objects 89.3% of the time, even though it had never seen those specific items before and didn't use any pre-trained data about how to grab them.

The Bottom Line

Think of this system not as a student memorizing a textbook, but as an artist who can look at a single photo of a new object and instantly sculpt a perfect 3D replica of it in their mind, complete with lighting and texture. Once that replica is built, the robot knows exactly how to interact with the real object.

The paper emphasizes that this is a "zero-shot" solution, meaning it works immediately on new things without the expensive and time-consuming process of collecting millions of training images first. It trades "big data" for "smart physics."

Technical Summary: Differentiable Inverse Graphics for Zero-shot Scene Reconstruction and Robot Grasping

Problem Statement

Robotic systems operating in novel, unstructured environments face significant challenges in perceiving and interacting with previously unseen objects. Current state-of-the-art approaches rely on large-scale neural networks trained on millions of labeled samples, creating high sample complexity and requiring expensive 3D data collection. Furthermore, many few-shot pose estimation methods still depend on extensive test-time data (e.g., hundreds of images or multiple 6D-annotated frames) to build scene representations. This reliance on massive datasets and test-time sampling hinders the deployment of data-efficient, interpretable, and generalizable robotic autonomy. Additionally, deep learning models often lack explainability, posing safety risks in high-torque applications where "hallucinations" can lead to physical injury. The core challenge is to reconstruct physically consistent scene parameters (meshes, lighting, materials, 6D poses) from a single RGBD image without prior 3D models or additional training data.

Methodology

The authors propose Differentiable Neuro-Graphics (DNG), a physics-grounded, sequential inverse optimization framework. The system solves a series of constrained optimization problems to infer scene properties from a single RGBD image and bounding box prompts. The pipeline consists of four distinct stages:

Object Detection and Segmentation:
The system first estimates bounding boxes using an object detector (e.g., EfficientDet or SSD). These boxes are fed into the Segment Anything Model (SAM), a foundation model, to generate object masks ( $\hat{M}_k$ ).
Robust Ellipsoid Estimation:
To initialize the 3D scene, the system projects the depth image into a point cloud and partitions it based on the segmentation masks. It then performs a Maximum A Posteriori (MAP) estimation to fit ellipsoidal primitives to each object. To handle sensor noise and over-segmentation, the authors employ robust prior distributions:
- Position: Modeled using a Laplace distribution.
- Depth Size: Modeled using a Log-normal distribution.
- Width/Height: Modeled using a Truncated Normal distribution to ensure physical plausibility (positive values).
  This step provides a robust initialization to avoid local minima in subsequent optimization.
Differentiable Scene Rendering (Sphere Optimization):
Using a custom differentiable renderer built with JAX primitives, the system optimizes scene parameters ( $\theta^s_k$ ) including object positions, scales, lighting conditions, and material properties (ambient, diffuse, specular, shininess, color).
- Loss Function: The optimization minimizes a weighted sum of mean absolute errors between rendered and observed images, depth maps, and masks.
- Constraints: Physical plausibility is enforced via barrier functions for material parameters (Phong reflection model boundaries). An optional 1D line constraint is used to reduce the search space for object positions, stabilizing convergence.
- Optimization: Solved using L-BFGS.
Differentiable Mesh Rendering:
The final stage refines the geometry from ellipsoids to triangular meshes. A control cage deforms the mesh vertices using mean value coordinates.
- Regularization: The loss function includes terms for mesh smoothness (discrete Laplacian), disparity smoothness, and a novel volume loss to ensure the mesh volume remains consistent with the initial spherical estimate.
- Optimization: Solved using ADAMW over 4,000 steps.
- Differentiable Masking: To address the non-differentiable nature of binary masks, the authors introduce a soft-mask function operating directly on rendered depth images, eliminating the need for pixel-mesh correspondence calculations.

Key Contributions

Zero-Shot Reconstruction Pipeline: A novel multi-stage optimization framework that reconstructs unseen objects from a single RGBD image without any 3D training data, multi-view captures, or 6D pose labels. It outperforms existing model-free few-shot pose estimation models on standard benchmarks.
Robust Optimization Methodology: The introduction of a probabilistic ellipsoid estimation using physically consistent priors (Laplace, Log-normal, Truncated Normal) to handle sensor noise. This serves as a critical initialization that prevents the optimizer from converging to local minima, a failure point observed in baseline methods.
Differentiable Graphics Engine: The development of a novel differentiable ray tracer for triangular meshes using only JAX primitives. This engine features a new differentiable soft-mask function and offers high portability and integration with deep learning libraries. It is significantly faster than existing path tracers (e.g., Mitsuba) while maintaining fidelity.

Results

The authors evaluated DNG on standard model-free few-shot benchmarks and a zero-shot robotic grasping task:

Pose Estimation Benchmarks:
- CLEVR-POSE: The robust ellipsoid initialization was shown to be critical; without it, a baseline failed to converge (0.008 ARV SD vs. 0.656 ARV SD).
- FewSOL, MOPED, and LINEMOD-OCCLUDED: DNG variants consistently outperformed state-of-the-art methods (Gen6D, OnePose++) in terms of ARV SD, Chamfer distance, and Hausdorff distance. Notably, on the LINEMOD-OCCLUDED dataset, the zero-shot approach ranked among the top 100 entries on the BOP Challenge leaderboard, competing against fully supervised models trained on massive datasets.
- Resolution Trade-off: Higher input resolutions yielded better reconstruction accuracy at the cost of increased optimization time.
Zero-Shot Robot Grasping:
- The reconstructed scenes were used to plan grasps in simulation, which were then executed on a real UR5 robot with a Mia hand.
- Across 224 trials involving 10 diverse YCB objects (varying in size, weight, and transparency), the system achieved a 89.28% success rate.
- This performance is comparable to data-driven approaches like GraspSAM (86% success) but achieved without any grasp-specific training data or pre-training on millions of labeled grasps.

Significance and Claims

The paper claims that DNG offers a pathway toward more data-efficient, interpretable, and generalizable robot autonomy. By combining foundation models for segmentation with physics-based differentiable rendering, the approach enables robots to operate in novel environments without the prohibitive cost of collecting large 3D datasets or performing extensive test-time sampling.

The authors emphasize that their method addresses the "sample complexity" and "explainability" hurdles inherent in deep learning. By solving an inverse optimization problem grounded in physical laws (lighting, material, geometry), the system provides a transparent mechanism for scene understanding, reducing the risk of "hallucinations" that could lead to physical harm in robotic applications. While the current optimization time is approximately one minute on an RTX 5090 GPU, the work demonstrates the feasibility of zero-shot, physically consistent scene reconstruction and manipulation.