💻 computer science

Generative Blocks World: Moving Things Around in Pictures

The paper introduces Generative Blocks World, a method that enables intuitive editing of generated images by manipulating 3D geometric primitives to guide a flow-based image synthesis process, thereby achieving superior visual fidelity, object identity preservation, and editability compared to existing techniques.

Original authors: Vaibhav Vavilala, Seemandhar Jain, Rahul Vasanth, D. A. Forsyth, Anand Bhattad

Published 2026-03-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Vaibhav Vavilala, Seemandhar Jain, Rahul Vasanth, D. A. Forsyth, Anand Bhattad

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a photograph of a messy living room. You want to move the cat from the sofa to the floor, or zoom the camera in so the cat looks bigger, or even rotate the room so you see the back of the sofa.

If you try to do this with standard photo editing software (like Photoshop), it's like trying to rearrange a 2D painting. You have to cut out the cat, paste it somewhere else, and hope the shadows and background look right. If you try to zoom in, the cat just gets blurry and pixelated; it doesn't actually look like it's getting closer.

"Generative Blocks World" is a new tool that changes the rules. Instead of treating the image as a flat picture, it treats the image like a 3D Lego set.

Here is how it works, broken down into simple steps:

1. The "Magic Scan" (Turning Pictures into Blocks)

When you upload a photo, the AI doesn't just see pixels. It instantly scans the image and breaks it down into a collection of 3D geometric blocks (like cubes, pyramids, and weirdly shaped chunks).

The Analogy: Imagine looking at a clay sculpture and realizing it's actually made of hundreds of small, distinct Lego bricks glued together. The AI figures out which brick is the cat, which is the table, and which is the wall.

2. The "Playground" (Moving the Blocks)

Because the image is now a collection of 3D blocks, you can interact with it just like a toy.

Moving Objects: You can grab the "cat block" and drag it to the left. Because the AI knows it's a 3D object, it automatically adjusts the perspective. If you move the cat closer to the "camera," the cat gets bigger, just like in real life.
Changing the Camera: You can tell the AI to "zoom in" or "move the camera to the right." The blocks shift accordingly, revealing parts of the scene that were previously hidden (like the back of the sofa) and hiding parts that were visible.
Scaling: You can grab just the "head block" of the cat and make it huge, while the body stays normal size.

3. The "Painter" (Re-imagining the Scene)

Once you've moved your blocks around, the AI acts like a master painter. It looks at your new arrangement of blocks and asks, "Okay, what would this scene look like if it were a real photo?"

The Texture Hint: This is the secret sauce. When the AI paints the new picture, it doesn't just guess what the cat's fur should look like. It remembers the original cat's fur and "projects" it onto the new position of the block.
The Result: The cat moves, but its fur pattern stays exactly the same. The shadows shift correctly. The background fills in naturally where the cat used to be.

Why is this better than other tools?

Drag-and-Drop vs. 3D Blocks: Other tools let you drag a point on a 2D screen. It's like trying to move a car in a video game by dragging a sticker on the screen. Sometimes the car stretches weirdly, or the wheels disappear. This tool moves the actual 3D car, so the wheels stay round and the car stays solid.
No "Training" Needed: Many AI tools need to be taught specifically how to move objects, which takes a long time. This tool uses a pre-trained "painter" and just gives it a new set of instructions (the blocks) on the fly. It's like having a painter who can instantly adapt to any new layout you give them.

The Limitations (The "Glitches")

Like any new technology, it's not perfect.

Lighting: If you move a block, the AI might not perfectly calculate how the sunlight hits it from a new angle (like a new shadow).
Complex Shapes: If an object is very weird (like a coffee mug handle), the "blocks" might not fit perfectly, and the AI might get confused about where the handle goes.

In Summary

Generative Blocks World turns a flat, static photo into a playable 3D world made of building blocks. You can rearrange the furniture, move the pets, and change the camera angle, and the AI will instantly repaint the scene to look like a brand-new, realistic photograph that respects the laws of physics and perspective. It's the difference between editing a drawing and playing with a 3D model.

1. Problem Statement

Current image editing methods, particularly those based on diffusion models, often struggle with 3D consistency and geometric control.

Limitations of 2D Interfaces: Methods like DragGAN or DragDiffusion operate in 2D pixel space. They lack explicit 3D understanding, leading to artifacts when objects are moved (e.g., incorrect scaling, shape distortion, or loss of texture identity) and making camera movements difficult to control.
Limitations of Explicit 3D Reconstruction: Methods that reconstruct full 3D meshes (e.g., Image Sculpting, OMG3D) are often bottlenecked by the quality of the initial reconstruction and require complex, multi-stage pipelines that are not robust to "in-the-wild" images.
The Gap: There is a need for an editing paradigm that offers explicit 3D control (shape constancy, contact consistency, shape completion) while remaining compact and editable, without requiring heavy fine-tuning of generative models or perfect 3D mesh reconstruction.

2. Methodology: Generative Blocks World

The proposed method decomposes a scene into a sparse set of convex 3D primitives (polytopes) which serve as an editable intermediate representation. The pipeline consists of four main stages:

A. Primitive Extraction (Convex Decomposition)

Representation: Scenes are represented as assemblies of blended 3D convex polytopes (based on CVXNet). Each primitive is defined by a set of half-planes.
Learning: A lightweight encoder-decoder network (ResNet-18 + FC layers) predicts the parameters of these primitives from an input RGB-D image.
Refinement: The initial prediction is refined using gradient-based optimization to align the primitives with the observed depth map (supervised by DepthAnythingv2).
Variable Scale: The system supports variable numbers of primitives ( $K$ ), allowing users to choose between coarse edits (few primitives) and fine-grained edits (many primitives).

B. Depth-Conditioned Generation

Base Model: The system utilizes a pretrained Rectified Flow (RF) model (specifically FLUX.1 Depth), which is conditioned on depth maps.
Rendering: The edited 3D primitives are rendered into a depth map from the desired camera viewpoint. This depth map serves as the primary geometric condition for the generative model.
No Fine-Tuning: Unlike methods like LooseControl, this approach does not require fine-tuning the diffusion model on primitive-based data; the primitives are accurate enough to work with off-the-shelf depth-conditioned models.

C. Texture Hint Generation (Core Innovation)

To preserve object identity and texture during edits (especially camera moves or object translations), the authors introduce a projection-based texture hint:

Point Cloud Correspondence: The system establishes correspondences between 3D points in the original view and the edited view using the primitive transformations (translation, rotation, scaling).
Hint Image ( $x_{hint}$ ): Pixels from the source image are projected onto the new view based on these correspondences. This creates a rough approximation of what the edited image should look like.
Confidence Mask ( $m$ ): A mask is generated to indicate regions where the hint is reliable (e.g., visible surfaces) and where it is not (e.g., occlusions, depth discontinuities, or boundaries).
Integration: During the diffusion denoising process (specifically within a defined timestep window $[t_{start}, t_{end}]$ ), the latent representation is updated as a weighted combination of the noised hint and the current latent:
$x_t = (1 - m) \cdot x_{hint, t} + m \cdot x_t$
This ensures the generated image respects the new geometry while retaining the source texture where the hint is valid.

D. Editing Workflow

Users interact with the scene by manipulating the 3D primitives (moving, scaling, rotating, or deleting them) and adjusting the camera. The system then regenerates the image conditioned on the new depth map and the generated texture hint.

3. Key Contributions

Generative Blocks World Pipeline: A novel framework fusing convex primitive abstractions with flow-based generative models to enable natural 3D interaction for image editing.
Texture-Hint Procedure: A training-free algorithm that generates projection-based texture hints and confidence masks. This allows for accurate camera moves and object translations while preserving object identity and high-frequency texture details, outperforming key-value injection methods (like StableFlow) in geometric scenarios.
Variable Resolution Control: The ability to decompose scenes into varying numbers of primitives ( $K$ ), enabling both coarse structural edits and fine-grained part-level manipulations within the same framework.
Training-Free Synthesis: The method leverages existing, powerful depth-conditioned models (FLUX) without requiring retraining or LoRA fine-tuning for the specific primitive representation, provided the primitives are accurate.

4. Results and Evaluation

The authors evaluated their method against state-of-the-art baselines (DragDiffusion, LooseControl, StableFlow) on metrics of geometric consistency and texture retention.

Geometric Consistency:
- Metric: Absolute Relative Error (AbsRel) between the requested primitive depth and the synthesized image depth.
- Result: Generative Blocks World achieved significantly lower AbsRel errors (0.072 vs. 0.143 for LooseControl), demonstrating superior adherence to the requested geometry.
Texture Consistency:
- Metric: PSNR and SSIM after warping the edited image back to the original view.
- Result: The method achieved much higher PSNR (18.7 vs. 6.65) and SSIM (0.874 vs. 0.670) compared to LooseControl.
Qualitative Comparisons:
- Camera Moves: Unlike LooseControl, which often hallucinates new objects or changes scene content (e.g., changing the number of apples) during camera pans, Generative Blocks World maintains scene consistency.
- Object Manipulation: Compared to DragDiffusion, which struggles with scaling and shape preservation, the primitive-based approach maintains correct perspective and object identity.
- Failure Cases: The method struggles with non-convex shapes (e.g., chair undersides), large rotations (>50°), and complex lighting effects (shadows/reflections) that are not captured by the primitive geometry.

5. Significance

Bridging 3D and Generative AI: The paper successfully bridges the gap between explicit 3D scene representations and modern generative models. It proves that simple, convex geometric abstractions are sufficient to drive high-fidelity image synthesis.
Intuitive 3D Editing: It offers a more intuitive interface for 3D editing than point-dragging (which is ambiguous) or full mesh reconstruction (which is brittle). Users can "move things around" with predictable 3D consequences (scaling, occlusion, perspective).
Efficiency: By avoiding the need to train new diffusion models for specific primitive types and using off-the-shelf depth conditioners, the approach is computationally efficient and adaptable to various scenes.
Future Direction: The work suggests a path toward "Generative Blocks World" as a standard for controllable image editing, where the geometric representation acts as the primary control signal for generative synthesis.