💻 computer science

SceneForge: Structured World Supervision from 3D Interventions

SceneForge is an intervention-driven framework that generates structured, consistent multimodal supervision by modeling scenes as editable 3D worlds with semantic, geometric, and physical dependencies, thereby enabling the creation of aligned counterfactual and multi-view data that improves performance in object and scene removal tasks.

Original authors: Jizhizi Li, Jiayang Ao, Danny Wicks, Petru-Daniel Tudosiu

Published 2026-05-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jizhizi Li, Jiayang Ao, Danny Wicks, Petru-Daniel Tudosiu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to edit photos. If you only show the robot a "before" picture and an "after" picture, it has to guess what happened in between. Did the shadow move because the light changed? Did the background appear because an object was moved? Without knowing the "rules" of the 3D world, the robot often makes mistakes, like leaving a floating shadow behind or filling in the background with the wrong texture.

SceneForge is a new system designed to fix this by teaching the robot using a 3D virtual world instead of just flat pictures.

Here is how it works, using simple analogies:

1. The "Lego World" vs. The "Photograph"

Most current AI training data is like a stack of photographs. You have a photo of a room, and another photo of the same room with a chair removed. The AI has to guess how the floor looks under the chair.

SceneForge is different. It builds a digital Lego world.

In this world, every object (a chair, a table, a wall) is a real 3D piece with rules.
The system knows that the chair sits on the floor, casts a shadow, and might reflect light.
When you want to "edit" the scene, you don't just paint over the chair in a photo. You physically pick up the Lego chair and remove it from the world.

2. The "Domino Effect" (Interventions)

The paper calls this an "intervention." Think of it like knocking over a row of dominoes.

Old way: You erase a domino from a picture. The space where it was is just blank.
SceneForge way: You remove the domino. Because the system knows the physics of the world, it automatically updates everything else.
- The floor underneath the domino is revealed.
- The shadow the domino was casting disappears.
- The reflection in a nearby mirror changes.
- The lighting adjusts to fill the empty space.

Because the system updates the entire world state at once, every single result (the new floor, the new shadow, the new reflection) is perfectly consistent with the others. They are all "aligned" by the rules of the 3D world, not guessed by the AI.

3. The "Master Blueprint"

The authors created a massive library of these 3D worlds (over 2,000 rooms) using tools like Infinigen and Blender.

They didn't just take pictures; they built a master blueprint for every scene.
From this one blueprint, they can generate:
- The "before" photo.
- The "after" photo (with an object removed).
- The "mask" (showing exactly what was removed).
- The "shadow layer" (showing just the shadows).
- Views from different angles (like looking at the room from the ceiling or the corner).

All of these different views and layers come from the same single source of truth. This means the AI training data is perfectly synchronized.

4. The Results: A Better Student

The researchers tested this by teaching an AI to remove objects from images. They compared three students:

Student A: Trained on 30,000 standard "before/after" photo pairs found online.
Student B: Trained on a mix of those online photos and some SceneForge data.
Student C: Trained only on SceneForge data (fewer total images, but higher quality).

The Outcome:
Student C (trained only on the "Lego world" data) performed the best. Even though they saw fewer examples, they learned the rules of how shadows and backgrounds behave better than the student who saw thousands of messy, unconnected photos.

When asked to remove a chair, Student C correctly removed the shadow underneath it.
Student A often left the shadow behind or filled the background with the wrong color.

Summary

SceneForge is a tool that stops AI from guessing how the world works. Instead of showing it a pile of disconnected photos, it gives it a playable 3D world. By letting the AI practice removing objects in this virtual world, the AI learns to handle complex effects like shadows, reflections, and hidden backgrounds naturally. This leads to much smarter and more realistic image editing.

Technical Summary: SceneForge

Problem Statement

Modern multimodal learning systems require supervision that is not only large-scale but also structured, controllable, and consistent across edits, viewpoints, and modalities. Existing datasets typically provide observation-level snapshots (e.g., before-and-after image pairs) that expose rendered outputs rather than the underlying scene state. Consequently, these datasets fail to provide matched interventions, physically consistent counterfactuals, layered decompositions, or aligned multimodal signals for the same scene. While diagnostic synthetic datasets (e.g., CLEVR) offer controllable generation, they focus on fixed reasoning tasks rather than intervention-driven supervision for generative and editing models. Similarly, existing synthetic 3D pipelines often treat scenes as static rendering configurations rather than persistent worlds where state transitions can be manipulated to induce consistent updates across entities and views.

Methodology: The SceneForge Framework

SceneForge is a world-centric framework that generates structured supervision by treating scenes as editable world states rather than collections of independent images. The pipeline proceeds through four main stages:

1. World State Representation

Instead of flat rendering configurations, SceneForge represents a scene at time $t$ as a structured tuple $w_t = (S_t, D_t, A_t, R_t, E_t)$ :

$S_t$ (Structural Elements): Stable components like floors, walls, and fixed geometry.
$D_t$ (Dynamic Objects): Entities subject to intervention (insertion, removal, relocation).
$A_t$ (Attributes): Semantic, geometric, and physical properties (category, pose, material, support status).
$R_t$ (Relations): Dependencies such as support, containment, occlusion, and visibility.
$E_t$ (Environmental Factors): Lighting and rendering conditions.

This representation ensures the scene is treated as a persistent system where changes to dynamic objects propagate through structural and relational constraints.

2. Intervention Model

An intervention $\iota$ is defined as an explicit operation on dynamic objects or environment variables (e.g., object removal, lighting modification). The system applies a provisional edit $\tilde{w}_{t+1} = I(w_t, \iota)$ . Crucially, this stage includes scene-querying modules to verify validity, ensuring edits are physically plausible (e.g., objects placed on valid support surfaces, no collisions) and semantically coherent before rendering.

3. Dependency-Aware State Propagation

To handle non-local consequences of local edits, SceneForge applies a propagation operator $P$ to update the world state: $w_{t+1} = P(\tilde{w}_{t+1})$ . This step recomputes visibility, occlusion, support relations, and lighting effects (shadows, reflections) based on the new state. This ensures that all downstream supervision signals remain semantically and physically consistent with the edited world, distinguishing the approach from naive "before-and-after" image generation.

4. Multimodal Supervision Generation

From the consistent world state transition, the framework renders aligned supervision across multiple modalities ( $y_m = G_m(w_t, w_{t+1}, c)$ ). Because all outputs derive from the same scene graph, they are aligned by construction. Generated outputs include:

RGB observations and multi-view renderings.
Counterfactual edits (object removal).
Decomposition layers (object masks, amodal completions).
Effect-aware channels (shadows, reflections, lighting).
Semantic and relation annotations.

Dataset Instantiation

The authors instantiated SceneForge using Infinigen and Blender to create a licensing-clean resource. The dataset includes:

SceneForge (S): A single-camera corpus with 2,369 rooms and 25,688 object-linked layers.
SceneForge (M): A multi-camera corpus with 517 rooms and 2,772 registered multi-view instances under 8 camera settings.
Derived Tasks: Filtered datasets for object removal (SceneForge-Removal) and scene-level structural removal (SceneForge-IndoorDynamicRemoval), totaling over 13,000 single-view and 4,000 multi-view edited instances.

Empirical Results

The framework was evaluated on object-level and scene-level removal tasks using the Qwen-Image-Edit-2509 backbone with LoRA fine-tuning. Three training settings were compared under matched budgets:

OpenData-30K: 30K public training samples.
OpenData+SceneForge-30K: Public data mixed with SceneForge samples.
SceneForge-16K: Only 16K SceneForge samples.

Key Findings:

Performance Gains: Models trained with SceneForge supervision (OpenData+SceneForge-30K) significantly outperformed those trained solely on public data (OpenData-30K) across all benchmarks (OpenData-30K, RORem, RemovalBench) and metrics (PSNR, SSIM, LPIPS).
Data Efficiency: The SceneForge-16K setting achieved the best results on every metric across all four test sets, despite using fewer total samples than the 30K settings. This suggests that high-quality, intervention-consistent supervision outweighs larger, mixed training pools.
Qualitative Improvements: In hard-case scenarios involving complex shadows and reflections (Removal-HardEffects), SceneForge-trained models demonstrated superior ability to remove associated visual side effects and restore consistent backgrounds compared to baseline models.
Multi-View Consistency: In the multi-view setting (SceneForge-Removal M), the framework maintained spatial consistency across diverse viewpoints (high-overlap and diverse angles) because all renders were derived from the same persistent 3D world state.

Significance and Claims

The paper claims that SceneForge provides a practical and scalable foundation for intervention-consistent multimodal learning. By modeling supervision as structured state transitions in editable worlds, the framework solves the alignment problem inherent in observation-level datasets. The authors assert that this approach:

Generates aligned outputs (counterfactuals, decompositions, effects) without post-hoc image-space matching.
Offers a flexible engine that can extend to richer linguistic supervision and broader asset diversity.
Demonstrates that "cleaner, more intervention-consistent supervision" can yield superior downstream utility compared to larger, less structured datasets.

The authors position SceneForge not as a replacement for real or hybrid datasets, but as a complementary engine that addresses the specific need for consistency under edits, viewpoints, and scene-level interventions.