Ctrl&Shift: High-Quality Geometry-Aware Object Manipulation in Visual Generation

1. Problem Statement

Object-level manipulation—relocating or reorienting objects within images or videos while preserving scene realism—is a critical task for film post-production, augmented reality (AR), and creative editing. However, existing methods face a fundamental trade-off:

• Geometry-based approaches (e.g., NeRF, 3D Gaussian Splatting) offer precise control over viewpoint and geometry but suffer from poor generalization, often requiring explicit 3D reconstruction, synthetic data, or per-scene optimization.
• Diffusion-based approaches generalize well to real-world "in-the-wild" content but lack fine-grained geometric control, leading to warped objects, inconsistent shadows, and background artifacts when viewpoint changes are required.

The core challenge is achieving geometry-consistent object manipulation with strong generalization to real-world content without relying on explicit 3D modeling at inference time.

2. Methodology: Ctrl&Shift

The authors propose Ctrl&Shift, an end-to-end diffusion framework that injects precise viewpoint control directly into a 2D diffusion process. The method decomposes manipulation into two sub-tasks: object removal and reference-guided inpainting under explicit camera pose control.

A. Core Architecture

• Backbone: Based on a ControlNet-style Diffusion Transformer (DiT) with a hidden size of 1536, built upon the Wan-1.3B video generation model.
• Input Conditioning: The model processes a unified latent space containing:
  • Source frames ($X_{src}$) and a reference object image ($I_{ref}$).
  • Source mask ($M_{src}$) and an estimated target mask ($\hat{M}_{tgt}$).
  • Relative Camera Pose Descriptor ($f$): An 8-dimensional vector encoding relative rotation (axis-angle), translation, and normalized device coordinate (NDC) shifts.
• Encoding Mechanisms:
  • Masks: Instead of VAE encoding, masks are mapped to the latent grid via a space-to-depth (pixel-unshuffle) operation to preserve binary semantics and stride alignment.
  • Camera Pose: The relative pose $f$ is encoded via Fourier positional encoding and a lightweight MLP, then injected into the DiT blocks via cross-attention to guide the viewpoint while preserving object identity.

B. Multi-Task, Multi-Stage Training Strategy

To disentangle background preservation, object identity, and geometric transformations, the authors employ a specific training regimen:

1. Multi-Task Learning:

   • Main Task: Full object manipulation (move object, change pose, keep background).
   • Auxiliary Task 1 (Object Removal): Removes the object and inpaints the background (reference set to white, target mask to zero).
   • Auxiliary Task 2 (Reference Inpainting): Synthesizes the reference object onto a background with specific camera control (no removal required).
   • Goal: This prevents the model from conflating signals, ensuring it learns to handle identity, background, and geometry independently.

2. Two-Stage Training:

   • Stage I (Object Prior & Pose Learning): Pretrained on a large-scale synthetic dataset (~2M pairs) of 3D meshes rendered against white backgrounds with randomized camera poses. This establishes robust geometric priors and category-agnostic object understanding.
   • Stage II (Background Preservation & Fine-Tuning): Fine-tuned on a curated dataset of high-quality real-world images and videos (100K pairs). The main backbone is frozen; only the conditioning control branch is updated to learn complex scene harmonization and photorealism.

C. Data Construction Pipeline

A scalable pipeline generates paired training data from real-world content:

1. Reconstruction: Uses Hunyuan3D-2 to reconstruct watertight, textured meshes from source images.
2. Pose Estimation: Uses differentiable rasterization to estimate the source camera pose by aligning the mesh silhouette to the object mask (filtering for IoU $\ge$ 0.90).
3. Synthesis: Renders the object under a novel target camera pose and uses a learned Object Pasting Model (a reference-image inpainting network) to seamlessly composite the rendered object onto the background, handling lighting and shadow harmonization.

3. Key Contributions

1. Conceptual Innovation: The first framework to unify fine-grained geometric control and real-world generalization for object manipulation without explicit 3D modeling at inference. It shifts the paradigm from "lifting to 3D" to "injecting 3D control into 2D diffusion."
2. Architectural Design: A multi-task, multi-stage training strategy that effectively disentangles background, identity, and pose signals, enabling coherent responses to multiple conditioning inputs.
3. Data Pipeline: A novel method for constructing real-world paired supervision with accurate camera control by combining 3D reconstruction, differentiable pose estimation, and learned harmonization.
4. Benchmarking: Introduction of GeoEditBench, a new benchmark specifically designed for geometry-aware editing, featuring 346 carefully curated image pairs with ground-truth camera poses.

4. Experimental Results

The model was evaluated on ObjectMover-A and the new GeoEditBench, outperforming state-of-the-art methods including DragAnything, 3DiT, VACE, Flux-Kontext, and Nano-Banana.

• Quantitative Performance:
  • GeoEditBench: Achieved the highest PSNR (28.71), DINO (85.23), and CLIP (90.44) scores, indicating superior background preservation and object identity retention.
  • Geometric Accuracy: Achieved the lowest Pose MAPE (17.70%) and highest Object IoU (0.83), demonstrating precise control over camera pose and object placement.
  • ObjectMover-A: Outperformed competitors with a PSNR of 28.69 and a DreamSim score of 0.075 (lower is better).
• Qualitative Results: Visual comparisons show Ctrl&Shift maintains background consistency and handles complex viewpoint changes (yaw, pitch, distance) without the artifacts (warping, identity drift) seen in other methods.
• Ablation Studies: Confirmed that removing Stage 1 severely degrades geometric fidelity (Pose MAPE jumps to 32.50%), while removing Stage 2 reduces visual quality (PSNR drops to 24.83).

5. Significance and Limitations

Significance:
Ctrl&Shift bridges the gap between the structural rigor of geometry-based pipelines and the flexibility of generative diffusion. It enables professional-grade, controllable object editing in real-world scenarios without the computational overhead of 3D reconstruction during inference, making it highly applicable for AR, film editing, and creative tools.

Limitations:

• User Interface: The current 8D relative pose descriptor is not user-friendly; future work aims to map intuitive interactions (e.g., 2D mouse drags) to this descriptor.
• Physical Realism: The model relies on data-driven learning for lighting and shadows, which can struggle with drastic illumination changes or complex physical interactions (e.g., reflections).
• Object Constraints: The pipeline assumes rigid geometry. It struggles with non-rigid objects (cloth, hair), transparent materials (due to texture baking errors in 3D reconstruction), and complex occlusions where an object moves behind another scene element.
• Video Fidelity: While effective for images, video manipulation currently exhibits lower fidelity due to the inability to capture non-rigid deformations in the rigid 3D reconstruction pipeline.

In conclusion, Ctrl&Shift represents a significant step forward in controllable visual generation, offering a scalable solution for geometry-aware object manipulation that balances precision with real-world applicability.