💻 computer science

PhyEdit: Towards Real-World Object Manipulation via Physically-Grounded Image Editing

The paper introduces PhyEdit, a framework that enhances real-world object manipulation in image editing by leveraging explicit 3D geometric simulation and joint 2D-3D supervision, supported by a new dataset (RealManip-40K) and benchmark (ManipEval) to achieve superior physical accuracy and spatial consistency compared to existing models.

Original authors: Ruihang Xu, Dewei Zhou, Xiaolong Shen, Fan Ma, Yi Yang

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ruihang Xu, Dewei Zhou, Xiaolong Shen, Fan Ma, Yi Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are playing with a digital photo editor, the kind where you can drag a cat from the left side of a picture to the right. Most of these tools work like a flat sticker: they just slide the image around on a 2D screen. But the real world isn't flat; it's deep. If you move a toy car closer to you in real life, it doesn't just slide sideways; it gets bigger, its details sharpen, and it might block the view of things behind it. This is the tricky part of "3D manipulation" in computer science. Scientists are trying to teach computers to understand this depth, perspective, and the laws of physics so that when they edit a picture, the objects behave like real things, not just flat stickers. This is crucial because if we want computers to help robots move things or to build "world models" that simulate reality, the computer needs to know that an object can't just float through a wall or change size without a reason.

Enter PhyEdit, a new tool developed by researchers at Zhejiang University that tries to solve this "flat sticker" problem. Think of PhyEdit as a digital sculptor who doesn't just paint on a canvas but first builds a 3D model of the scene in their head. When you ask PhyEdit to "move this banana to the left," it doesn't just slide the pixels. Instead, it uses a special "3D foundation model" to guess how deep the banana is, lifts it out of the flat picture into a virtual 3D space, moves it exactly where you want, and then projects it back down. Crucially, it checks its own work using a "joint supervision" system, which is like a teacher grading both the final drawing (the 2D image) and the 3D blueprint (the depth map) to make sure the object didn't magically shrink or stretch in a weird way.

The researchers found that while existing tools often mess up the size or position of objects when they try to move them in 3D, PhyEdit is much better at keeping things physically accurate. To prove this, they didn't just guess; they built a massive new dataset called RealManip-40K, which contains 40,000 pairs of real-world photos showing objects being moved in 3D space, complete with depth measurements. They also created a test called ManipEval to grade how well different AI models handle these moves. When they ran the tests, PhyEdit beat out many other methods, including some very powerful commercial models that cost money to use. For instance, on a specific test measuring how close the object's new position was to the target, PhyEdit scored a 65.33 (on a scale of 0 to 100), while the next best commercial model, Nano Banana Pro, scored 59.97. In terms of 3D shape accuracy (Chamfer distance), PhyEdit achieved a score of 18.93, significantly lower (which is better) than the commercial model's 25.33.

The paper suggests that this approach works because it combines a "plug-and-play" 3D guide with a training method that looks at both the 2D image and the 3D depth simultaneously. However, the authors are careful to note that this isn't a perfect magic wand yet. The system still struggles if the 3D map it builds is wrong (like if the depth sensor gets confused by a shiny object) or if the request is too extreme, like moving an object so close to the camera that it would disappear behind the lens. They also found that while PhyEdit is great at moving things, it's not necessarily better at changing the object's color or texture unless you give it very specific instructions. Ultimately, the paper suggests that by grounding image editing in real physics and 3D geometry, we are taking a significant step toward creating AI that can truly understand and manipulate the world, rather than just rearranging pixels on a flat screen.

Technical Summary: PhyEdit

Problem Statement

Existing visual generative models, including advanced diffusion transformers, struggle to perform physically accurate object manipulation in image editing. While these models excel at 2D-level edits or follow detailed coordinate instructions, they often fail to adhere to 3D physical laws, resulting in incorrect scaling, inconsistent trajectories, and implausible depth variations during object movement. This limitation stems from two primary gaps:

Lack of Explicit 3D and Physical Control: Current models lack mechanisms to incorporate perspective projection laws, leading to physically implausible behaviors (e.g., objects not resizing correctly as they move closer or further away).
Absence of High-Quality Real-World Data and Benchmarks: Existing datasets focus on 2D changes or synthetic 3D assets, lacking real-world samples with clear physical laws (e.g., depth-induced size variations) and non-rigid object interactions. Furthermore, existing benchmarks primarily evaluate 2D metrics, overlooking depth accuracy and geometric consistency.

Methodology: PhyEdit

PhyEdit is a DiT (Diffusion Transformer)-based framework designed to frame 3D-aware editing as generative state rendering. It integrates explicit geometric simulation as contextual 3D-aware visual guidance.

1. Model Architecture

3D Transformation Module: Before the denoising process, the system takes a source image, an object mask, and a transition vector (target 3D position). It utilizes a 3D Foundation Model (e.g., Depth-Anything-3) to predict depth maps and camera poses. The object is then unprojected into 3D space, transformed according to the target vector, and re-projected to generate a depth-aware preview image.
DiT Backbone: The framework uses a pre-trained DiT (specifically Qwen-Image-Edit) as the backbone. The input token sequence includes text prompts, noisy latent tokens, source image tokens, and the transformed preview tokens. The preview acts as a plug-and-play contextual guide, providing explicit geometric priors without replacing the source image's native visual input.
Joint Supervision Strategy: To ensure physical accuracy, the training employs a joint loss function:
- 2D Level: Standard flow-matching loss on the latent space to ensure appearance reconstruction.
- 3D Level: A depth-space supervision term using Scale-Invariant Logarithmic (SILog) loss. The model decodes the denoised latent to an image, estimates its depth, and compares it against the target depth map. This forces the model to learn geometric correctness alongside appearance.

2. Dataset and Benchmark

To support this method, the authors introduce:

RealManip-40K: A high-quality real-world dataset containing 40,000 paired images ( $I_{src}, I_{tgt}$ ) demonstrating physical object manipulation in 3D space. It includes detailed depth annotations, object masks, and 3D coordinates. The data is curated from high-resolution videos (e.g., OpenVid-1M, VIDGEN-1M) using camera-static filtering and depth-aware selection to ensure significant depth changes and realistic interactions.
ManipEval: A specialized benchmark with 200 image pairs and ~320 objects, designed to evaluate 3D spatial control. It introduces multi-dimensional metrics including:
- 2D Spatial Accuracy: DIoU and Mask IoU.
- Depth Accuracy: AbsRel and $\delta_{1.25}$ .
- 3D Manipulation Accuracy: Chamfer Distance and Centroid Distance between reconstructed point clouds.
- Physical Plausibility: A VLM-based score (Phys-VLM) assessing lighting, shadows, occlusion, and global scene consistency.

Key Results

Extensive experiments on ManipEval demonstrate that PhyEdit outperforms existing baselines, including strong closed-source commercial models (e.g., Nano Banana Pro, GPT-Image-1.5, Qwen-Image-2.0-Pro).

Quantitative Performance: PhyEdit achieves the best overall performance across manipulation-related metrics. Notably, it improves DIoU by 5.36 points and reduces Chamfer distance by 6.40 points compared to Nano Banana Pro. It also achieves the highest Phys-VLM scores, indicating superior physical realism.
Multi-Object Manipulation: The advantage of PhyEdit is more pronounced in multi-object scenarios, where it maintains precise 3D geometry and consistent interactions better than baselines that often fail to edit all requested objects or produce inconsistent depth.
Ablation Studies:
- Depth Supervision: Pixel-level depth supervision (decoding to image then predicting depth) proves superior to latent-to-latent or latent-to-depth supervision, significantly improving 2D and 3D metrics.
- Reference Image: The inclusion of the 3D-transformed reference image (preview) is critical. Removing it causes a substantial drop in performance (e.g., Chamfer distance increases from 18.93 to 46.31), highlighting the difficulty of recovering fine-grained 3D intent from text alone.
Generalization: The model demonstrates strong generalization on continuous object manipulation trajectories and handles objects outside its training distribution (e.g., robotic arms).

Significance and Claims

The paper positions PhyEdit as a significant step toward building interactive world models capable of rendering 3D-aware and geometrically consistent state transitions.

Bridging the Gap: The work addresses the gap between 2D image editing and 3D physical simulation by introducing explicit geometric guidance and joint 2D-3D supervision.
Physical Grounding: Unlike previous methods that rely on implicit learning or heavy 3D reconstruction, PhyEdit achieves physically grounded editing through lightweight conditioning and training design, making it adaptable to different DiT-based editors.
Real-World Applicability: By providing a real-world dataset and a rigorous 3D benchmark, the authors aim to facilitate research in 3D image editing and object-centric robotic manipulation, where precise physical manipulation is essential.

The authors acknowledge limitations, such as the reliance on upstream geometry estimation (depth/camera/mask) which can propagate errors in extreme cases, and the current difficulty in handling highly complex, free-form text instructions for fine-grained constraints. However, they assert that their approach effectively narrows the performance gap between open-source models and leading closed-source systems in terms of physical plausibility and spatial control.