🤖 AI

Integrating Multimodal Large Language Model Knowledge into Amodal Completion

The paper proposes AmodalCG, a novel framework that leverages the real-world knowledge of Multimodal Large Language Models to selectively guide and refine the reconstruction of occluded object regions in amodal completion, significantly outperforming existing methods.

Original authors: Heecheol Yun, Eunho Yang

Published 2026-03-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Heecheol Yun, Eunho Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking at a photo of a delicious pizza, but a large slice of pepperoni is hiding behind a glass of soda. You can see the crust and the cheese on the left, but the right side is a mystery.

Amodal Completion is the computer's attempt to guess what that hidden part of the pizza looks like and "draw" it in so you can see the whole thing.

For a long time, computers were bad at this. They were like artists who only knew how to paint what they could see. If they tried to fill in the missing pizza, they might accidentally paint a whole new pizza, a sandwich, or just a weird blob because they didn't really understand what a pizza is or how big it should be.

This paper introduces a new method called AmodalCG. Think of it as hiring a super-smart detective (an MLLM) to help a painter (the image generator) finish the job.

Here is how it works, broken down into simple steps:

1. The "Do We Need Help?" Check (The Gatekeeper)

Imagine you are fixing a broken vase. If only a tiny chip is missing, you don't need a master architect to tell you how to fix it; you can just do it yourself. But if half the vase is gone, you need a plan.

The system first asks a small, fast AI: "Is this object mostly hidden, or can we just guess the rest?"

If the object is mostly visible: The system skips the expensive "detective" and just fills in the small gap. This saves time and money.
If the object is heavily hidden: The system calls in the Multimodal Large Language Model (MLLM). This is the "detective" that knows everything about the real world (e.g., "Buses have wheels at the bottom," "People have legs," "Pizza is round").

2. The Detective's Two Clues

Once the detective is called in, it doesn't just say "Draw a bus." It gives the painter two specific instructions:

Clue #1: The "How Big?" Map (Geometric Guidance)
Sometimes, the painter tries to fill in the missing part but makes the object huge, stretching it off the edge of the photo. The detective looks at the visible part and says, "Stop! The whole bus is only this big. Don't draw it any larger." It draws a precise invisible box around where the entire bus should be, so the painter knows exactly where to stop.
Clue #2: The "What Does It Look Like?" Description (Semantic Guidance)
The painter needs to know what to draw. If the detective just says "Bus," the painter might draw a red bus with a clown face. The detective gives a detailed description: "It's a blue city bus with a large windshield, side mirrors, and colorful stripes." This ensures the hidden part matches the visible part perfectly.

3. The "Try, Check, and Fix" Strategy

Even the best detective can make a guess that is slightly off. Maybe the detective thinks the bus is 10 feet long, but it's actually 12 feet.

To fix this, the system uses a Multi-Scale Strategy:

The detective gives three guesses for the size: Tight (small), Moderate (medium), and Coarse (large).
The painter tries to draw the bus using the Tight box first.
The Check: If the bus touches the edge of the box, the system knows, "Oh, the bus is bigger than this! The box was too small."
The system then tries the Moderate box. If that works, it stops. If not, it tries the Coarse box.

This ensures the final image is complete without accidentally painting things that shouldn't be there.

Why is this a big deal?

Previous methods were like a painter working blindfolded, guessing what's behind the curtain. This new method is like giving the painter a blueprint and a detailed description from an expert who has seen millions of buses, people, and animals.

The Result:

No more weird mistakes: The computer doesn't accidentally draw a chicken when it's supposed to draw a bus.
Better accuracy: It knows exactly how big the hidden object is.
Smarter: It only uses the "super-smart" AI when it's really necessary, making the process faster and cheaper.

In short, AmodalCG combines the artistic ability of image generators with the common sense of a super-smart AI detective to reconstruct hidden parts of the world perfectly.

1. Problem Statement

Amodal completion is the task of reconstructing the full appearance of an object, including parts that are occluded by other foreground objects, based solely on its visible (modal) parts.

Current Limitations: Existing methods (e.g., Stable Diffusion inpainting, pix2gestalt, MC Diffusion) rely heavily on the generative capabilities of visual models but lack real-world physical knowledge and common sense.
Specific Issues:
- Unnatural Outputs: Without explicit guidance, models often generate objects unrelated to the target or fail to infer the correct pose/state (e.g., sitting vs. standing).
- Mask Misalignment: Standard inpainting masks are often too large, leading to "extraneous generation" where the model fills areas outside the actual object boundary.
- Inefficiency: Existing approaches either ignore occlusion reasoning entirely or use Multimodal Large Language Models (MLLMs) only for segmentation, not for guiding the actual generation process.

2. Methodology: AmodalCG

The authors propose AmodalCG, a framework that selectively integrates MLLM knowledge to guide the amodal completion process. The pipeline consists of five key components:

A. Occluder Detection

Identifies occluders using semantic segmentation and a geometric order prediction network to generate an initial inpainting mask ( $M_{inpaint}$ ).

B. Guidance Decision Module (Selective Invocation)

Problem: Invoking a large MLLM for every sample is computationally expensive and unnecessary for lightly occluded objects.
Solution: A lightweight MLLM analyzes the isolated target object to determine if detailed guidance is needed.
- Output: A binary indicator (Need Guidance? Yes/No) and the object category.
- Strategy: If occlusion is minimal, the framework skips the heavy MLLM reasoning to save resources.

C. Geometric Guidance Module (Extent Estimation)

Goal: Predict the true bounding box of the full object (including occluded parts) to resize the inpainting mask.
Mechanism:
- If guidance is needed, the MLLM predicts three candidate bounding boxes at different scales: Tight, Moderate, and Coarse.
- Multi-scale Expansion: The inpainting module iteratively tests these scales, starting from the tightest. It stops when the generated object fits within the predicted boundary without touching the edge, ensuring the mask is neither too small (incomplete) nor too large (extraneous generation).
- Refinement: The final mask is the intersection of the predicted bounding box and the original inpainting mask.

D. Semantic Guidance Module (Content Reasoning)

Goal: Generate a detailed textual description of the hidden parts of the object to guide the diffusion model.
Innovation: To prevent the MLLM from describing the occluders (e.g., describing a rider when the rider is blocking the horse), the framework masks out the occluders in the visual input provided to the MLLM.
Output: A detailed prompt (e.g., "A brown cow standing on all four legs...") describing the occluded regions, which replaces generic category prompts.

E. Inpainting Module

Integrates the resized mask and the detailed text prompt into a Stable Diffusion (SD) inpainting model.
Uses the multi-scale expansion strategy to ensure the final output is complete and accurate.

3. Key Contributions

Novel Framework (AmodalCG): The first method to directly leverage MLLMs for reasoning about the extent and content of occluded regions to guide the generation process, rather than just for segmentation.
Dual Guidance Strategy:
- Geometric Guidance: Prevents over-extension and extraneous generation by dynamically resizing inpainting masks based on MLLM-predicted object extents.
- Semantic Guidance: Provides precise textual descriptions of hidden parts, overcoming the ambiguity of standard category-level prompts.
Efficiency & Robustness Strategies:
- Selective Invocation: Uses a lightweight model to skip MLLM reasoning for easy samples, reducing computational cost.
- Multi-scale Expansion: Mitigates MLLM uncertainty in size estimation by testing multiple candidate scales iteratively.
Occluder Masking: A specific technique to isolate the target object visually for the MLLM, preventing it from hallucinating descriptions of the occluding objects.

4. Experimental Results

The method was evaluated on COCO-A, BSDS-A, and MP3D-A datasets, as well as for Occluded Object Recognition (OOR).

Amodal Segmentation (mIoU):
- AmodalCG significantly outperforms baselines (pix2gestalt, MC Diffusion, OWAAC).
- Improvement: +5.49% mIoU on COCO-A compared to the best baseline.
- Robustness: Shows the most significant gains in "Hard" (high occlusion) scenarios, where previous methods fail to reconstruct the full object.
Occluded Object Recognition (Top-1 Accuracy):
- Improved by 2.92% over baselines.
- Demonstrates that the generated completions are semantically accurate enough to be correctly classified by CLIP.
Ablation Studies:
- Geometric Guidance: Proven to be the most critical component, drastically reducing extraneous generation.
- Semantic Guidance: Highly effective for OOR and complex poses, though less impactful on segmentation metrics for static objects.
- Multi-scale Strategy: Essential for handling the uncertainty of MLLM size predictions.

5. Significance

Bridging the Gap: This work successfully bridges the gap between visual generative models (which excel at texture but lack reasoning) and MLLMs (which possess rich common sense and physical knowledge).
Open-World Applicability: The method works for open-world objects without requiring additional training on specific amodal datasets, relying instead on the pre-trained knowledge of MLLMs.
Practical Impact: The improvements in reconstructing occluded objects are crucial for downstream applications in autonomous driving (e.g., seeing a pedestrian behind a car) and robotics, where understanding the full state of an object is vital for safety and interaction.
Efficiency: By introducing selective invocation, the paper demonstrates that high-performance reasoning can be achieved without the prohibitive cost of running large models on every single inference step.