Making Images Real Again: A Comprehensive Survey on Deep Image Composition

This paper presents the first comprehensive survey on deep image composition by systematically reviewing sub-tasks such as object placement, blending, harmonization, and shadow generation, while also introducing the `libcom` toolbox and an online workbench to unify and simplify these editing processes.

The Gist

Imagine you are a chef trying to create the perfect dish. You have a delicious piece of steak (the foreground) and a beautiful, steaming plate of mashed potatoes (the background). Your goal is to put the steak on the plate so it looks like it belongs there, ready to be served.

This paper is a massive "cookbook" and "toolkit" for a specific type of digital cooking called Image Composition. It's about taking an object from one photo and pasting it into another so that the final result looks like a real, single photograph, not a clumsy collage.

Here is the breakdown of the paper using simple analogies:

The Problem: The "Uncanny Valley" of Photos

When you just copy and paste a steak onto a plate, it often looks fake. It might look too big, the lighting might be wrong (the steak looks like it's in a dark cave while the potatoes are in bright sunlight), or it might be floating in mid-air.

The authors say these "fake" feelings come from three main problems:

1. Appearance: The colors and lighting don't match (e.g., a sunny steak on a rainy plate).
2. Geometry: The size or angle is wrong (e.g., a tiny elephant or a floating suitcase).
3. Semantics: It just doesn't make sense (e.g., a zebra in a living room).

The Solution: A Step-by-Step Assembly Line

The paper explains that fixing these issues isn't one big magic trick; it's a series of smaller tasks. Think of it like a factory assembly line for photos:

1. Object Placement (The "Where and How Big"):

   • The Job: Deciding where to put the steak and how big it should be.
   • The Analogy: It's like a game of Tetris. You need to find the perfect spot where the piece fits the puzzle. If you put a giant giraffe in a small room, it looks wrong. If you put a tiny car on a highway, it looks weird. The paper reviews tools that help the computer guess the right size and spot automatically.

2. Image Blending (The "Seamless Tape"):

   • The Job: Smoothing the edges so you don't see the cut lines.
   • The Analogy: Imagine cutting a picture out of a magazine with jagged scissors. If you tape it to a wall, you see the rough edges. Image blending is like using a magic eraser and a soft brush to blend the cut edges into the wall so no one can tell where the paper ends and the wall begins.

3. Image Harmonization (The "Lighting Match"):

   • The Job: Changing the color and brightness of the object to match the background.
   • The Analogy: If you take a photo of a red apple in a bright kitchen and paste it into a photo of a dark cave, the apple will look glowing and fake. Harmonization is like putting a dimmer switch on the apple to make it look like it's actually sitting in that dark cave.

4. Shadow & Reflection Generation (The "Grounding"):

   • The Job: Adding shadows or reflections so the object feels "heavy" and real.
   • The Analogy: If you put a ball on the floor, it casts a shadow. If you paste a ball into a photo without a shadow, it looks like it's floating. This step is like the computer drawing a shadow under the object so it feels like it's actually touching the ground.

The New Trend: The "Generative Chef"

For a long time, computers did these steps one by one (first place, then blend, then light). But recently, a new technology called Diffusion Models (the same tech behind AI art generators) has arrived.

• The Old Way: Like a human chef chopping, frying, and plating separately.
• The New Way (Generative Composition): Like a robot chef that looks at the ingredients and the plate and re-imagines the whole dish instantly. It doesn't just paste the steak; it redraws the steak to perfectly match the plate's lighting, texture, and shadows all at once. The paper notes this is the future, but it can sometimes be too "creative" and change the object too much.

The Toolkit: "Libcom"

The authors didn't just write a book; they built a toolbox called libcom.

• The Analogy: Imagine if instead of just reading a cookbook, you were given a fully stocked kitchen with every tool you needed (knives, pans, mixers) pre-installed. You just say "Make a composite," and the toolbox does the work. They also built a website where you can try these tools online.

The "Shopping" Alternative: Foreground Search

Sometimes, instead of trying to fix a bad cut-and-paste, the best solution is to find a better ingredient to begin with.

• The Analogy: If you have a picture of a living room and you want to add a chair, instead of trying to force a weird chair into the photo, you search a library of chairs to find one that is the exact right size, color, and style for that room. This is called Foreground Object Search.

Why Does This Matter?

This technology isn't just for fun. It helps with:

• Virtual Try-Ons: Seeing how a shirt looks on you without going to the store.
• Advertising: Putting a soda can on a beach scene for a commercial.
• Training AI: Creating fake photos to teach computers how to recognize objects (like teaching a self-driving car what a pedestrian looks like in the rain).

In summary: This paper is the ultimate guide to making digital photos look real. It maps out every step of the process, from finding the right spot for an object to painting the perfect shadow, and introduces new AI tools that are making this magic easier and more realistic than ever before.

Technical Summary

1. Problem Definition

Image Composition (also known as object insertion) is the task of combining a foreground object from one image with a background from another to create a realistic composite image. The core challenge lies in eliminating inconsistencies between the foreground and background, which otherwise make the result look artificial. The authors categorize these inconsistencies into three main types:

• Appearance Inconsistency: Includes unnatural boundaries (jagged edges), incompatible illumination statistics (lighting/shadows), missing or implausible shadows/reflections, and discrepancies in resolution, sharpness, or noise.
• Geometric Inconsistency: Includes unreasonable object size, lack of physical support (e.g., floating objects), unreasonable occlusion, and inconsistent perspectives.
• Semantic Inconsistency: Includes placing objects in semantically unreasonable contexts (e.g., a zebra in a living room) or unrealistic interactions with other objects.

The paper argues that the image composition task can be decomposed into specific sub-tasks, each targeting one or more of these inconsistencies.

2. Methodology and Sub-Task Breakdown

The survey systematically reviews existing deep learning methods across six key sub-tasks and one complementary task:

A. Object Placement

• Goal: Determine the reasonable scale, location, and shape of the foreground to satisfy geometric and semantic constraints.
• Methods:
  • Traditional: Rule-based approaches (e.g., scanline matching, support surface detection).
  • Deep Learning: Divided into Category-specific (predicting boxes for a class) and Instance-specific (predicting transformations for a specific object). Approaches include Generative (predicting transformations directly) and Discriminative (predicting a rationality score map for all possible locations). Recent trends favor dense discriminative approaches (e.g., FOPA) for efficiency and accuracy.

B. Image Blending

• Goal: Refine the boundary between foreground and background to eliminate jagged edges and color artifacts.
• Methods:
  • Traditional: Alpha blending, Laplacian pyramids, and Poisson blending (gradient domain consistency).
  • Deep Learning: Learnable networks that fuse features from foreground and background. Recent "mask-free" methods use bounding boxes to generate seamless blends without requiring precise initial segmentation masks.

C. Image Harmonization

• Goal: Adjust the foreground's illumination and color statistics to match the background (e.g., time of day, weather).
• Methods:
  • Rendering-based: Inferring 3D geometry and lighting (computationally expensive and error-prone).
  • Non-rendering-based: Color transfer, adversarial learning (GANs), and style transfer.
  • Deep Learning: End-to-end CNNs, attention mechanisms (separating foreground/background features), and Diffusion Models (recently achieving state-of-the-art results by modeling illumination distribution).
  • Variants: Includes Blind Harmonization (no mask provided) and Painterly Harmonization (matching realistic objects to artistic backgrounds).

D. Shadow & Reflection Generation

• Goal: Generate plausible shadows and reflections cast by the foreground onto the background.
• Methods:
  • Rendering-based: Estimating scene geometry and light sources to render shadows.
  • Deep Learning: Image-to-image translation using GANs and Diffusion Models. Diffusion models (e.g., SGDiffusion, RGDiffusion) have shown superior ability to handle complex shapes and lighting conditions compared to GANs.

E. Generative Image Composition

• Goal: A unified approach that performs blending, harmonization, shadow generation, and viewpoint adjustment in parallel using a single model.
• Methods: Primarily based on Diffusion Models (e.g., ControlNet, ObjectStitch). These models re-generate the foreground object within the context of the background rather than just pasting it, allowing for drastic viewpoint changes and better detail preservation.
• Types: Training-free (manipulating off-the-shelf models) vs. Training-based (fine-tuning on specific object-background pairs).

F. Foreground Object Search (FOS)

• Goal: Instead of inserting an arbitrary object, retrieve a compatible foreground from a library that naturally fits the background's illumination, geometry, and semantics.
• Methods: Deep learning encoders for feature extraction and compatibility scoring, often using teacher-student networks to distill knowledge from composite image discriminators.

3. Key Contributions

1. First Comprehensive Survey: This is the first survey dedicated specifically to deep image composition (object insertion), covering the entire pipeline from placement to final rendering.
2. Unified Taxonomy: The authors clarify the relationship between Inconsistencies (the problem), Issues (specific manifestations), Sub-tasks (the solution modules), and Pipelines (sequential vs. parallel execution).
3. Resource Aggregation:
   • Summarized existing datasets (e.g., OPA, iHarmony4, DESOBA, DEROBA) and evaluation metrics (FID, SSIM, User Studies).
   • Created libcom, the first open-source toolbox for image composition, assembling 10+ functions (blending, harmonization, shadow generation, etc.).
   • Developed an online workbench (libcom.ustcnewly.com) to demonstrate these capabilities.
4. Experimental Analysis: Provided comparative experiments across sub-tasks (e.g., comparing generative vs. discriminative object placement, GAN vs. Diffusion for harmonization) to guide future research.

4. Results and Findings

• Object Placement: Dense discriminative approaches (predicting score maps) generally outperform generative approaches in finding optimal locations, though they struggle with complex occlusions.
• Blending & Harmonization: Traditional gradient-domain methods (Poisson) can distort content. Deep learning methods, particularly those using attention mechanisms and diffusion models, offer better preservation of details while achieving seamless transitions.
• Shadow/Reflection: Diffusion-based methods significantly outperform GANs and traditional rendering in generating realistic shadows for complex shapes and lighting conditions.
• Generative Composition: Unified diffusion models are emerging as the state-of-the-art, capable of handling multiple sub-tasks simultaneously and adjusting viewpoints/poses, though they sometimes struggle with strict bounding box alignment.
• Datasets: A major bottleneck identified is the scarcity of high-quality, paired training data (composite + ground truth). Synthetic datasets (rendered) suffer from domain gaps, while real-world datasets are expensive to annotate.

5. Significance

• Research Roadmap: The paper provides a structured framework for researchers to understand the landscape of image composition, identifying gaps in current methods (e.g., handling transparent objects, complex occlusions, and semantic consistency).
• Practical Application: The introduction of libcom and the online workbench democratizes access to these technologies, enabling applications in:
  • E-commerce: Virtual try-on and furniture placement.
  • Advertising: Automatic product insertion into scenes.
  • Data Augmentation: Generating realistic synthetic data for object detection and segmentation tasks.
  • Entertainment & Art: Creating realistic visual effects and digital art.
• Future Direction: The authors highlight the shift from sequential pipelines to unified generative models (Diffusion) and the need for better datasets and evaluation metrics that account for semantic and geometric plausibility, not just pixel-level similarity.