AmodalSVG: Amodal Image Vectorization via Semantic Layer Peeling

1. Problem Statement

Current image vectorization methods operate under a modal perception paradigm, meaning they only trace visible pixels and ignore occluded regions. When converting raster images to Scalable Vector Graphics (SVG), these methods fail to handle occlusions effectively, resulting in:

• Geometric Incompleteness: Objects are split into disconnected fragments rather than being represented as cohesive entities.
• Semantic Entanglement: Vector primitives are mixed across different objects and backgrounds, making it impossible to isolate, edit, or manipulate individual objects without breaking the visual integrity.
• Limited Editability: The resulting SVGs cannot support object-level operations (e.g., repositioning, recoloring, or removing an object) because the underlying structure does not represent complete, independent semantic layers.

The paper introduces Amodal Image Vectorization, a new task aimed at reconstructing the complete geometry and appearance of objects, including their occluded regions, into semantically organized and editable vector layers.

2. Methodology

The authors propose AmodalSVG, a two-stage framework that performs semantic decoupling and completion in the raster domain before vectorization. This approach leverages the generative capabilities of Vision-Language Models (VLMs) and diffusion models to infer missing geometry, which is difficult to achieve directly in the vector domain.

Stage 1: Semantic Layer Peeling (SLP)

This stage iteratively decomposes the input image into semantically coherent, amodally complete raster layers.

• VLM-Driven Foreground Parsing: A Vision-Language Model (specifically Qwen3-VL) analyzes the image to reason about spatial overlaps and occlusion hierarchies. It identifies the "frontmost" object and outputs a semantic tuple (description and keywords).
• Grounded Segmentation: The VLM's keywords guide a GroundedSAM model to generate precise, pixel-accurate masks for the identified foreground object.
• Holistic Layer Completion:
  • Foreground Completion: If an object touches the image boundary, the canvas is expanded to synthesize the missing parts via inpainting.
  • Background Restoration: The identified foreground is "peeled" off, leaving a hole. A hybrid inpainting pipeline (combining LaMa for texture reconstruction and FLUX for semantic detail) fills this hole with the underlying background content. This ensures the remaining image is a structurally complete scene for the next iteration.
  • This process repeats until only the pure background remains, producing a stack of complete, independent raster layers.

Stage 2: Adaptive Layered Vectorization (ALV)

Once the complete raster layers are generated, each is vectorized individually into a compact set of SVG primitives.

• Content-Adaptive Initialization: Primitives are initialized based on the spatial and color properties of the specific layer mask, preventing random scattering and reducing the search space.
• Occlusion-Aware Primitive Pruning: Instead of relying on simple area or opacity heuristics, ALV calculates a visual contribution score for each primitive based on the global occlusion hierarchy. Primitives that are fully occluded by others are identified as redundant and pruned, preserving only those that contribute to the final visual output.
• Importance-Driven Primitive Addition: To capture fine details, new primitives are added dynamically based on a residual error map. The addition is guided by a temperature-scaled probability distribution that focuses on high-error regions (edges/textures).
• Error Budget Mechanism: The number of primitives to add is determined by a feedback loop that estimates the "cost-effectiveness" of adding primitives, balancing reconstruction fidelity with primitive efficiency.

3. Key Contributions

1. Amodal Image Vectorization Task: Defined a new task requiring the holistic reconstruction of complete object geometries (including occluded parts) from raster to vector formats.
2. Semantic Layer Peeling (SLP): A novel VLM-guided iterative strategy that decomposes scenes and uses hybrid inpainting to recover structurally complete, amodal appearances of objects.
3. Adaptive Layered Vectorization (ALV): A feedback-driven vectorization mechanism that dynamically manages primitive budgets via occlusion-aware pruning and importance-driven addition, ensuring high fidelity with minimal redundancy.
4. Object-Level Editability: The resulting SVGs consist of semantically separated, amodally complete layers, enabling direct editing (reposition, recolor, resize, remove, reorder, replace) in the vector domain without artifacts.

4. Experimental Results

The authors evaluated AmodalSVG against state-of-the-art methods (e.g., DiffVG, LIVE, LayerTracer, AdaVec) on 30 diverse real-world images.

• Visual Fidelity: AmodalSVG achieved superior quantitative metrics, including the lowest MSE (0.00072), highest PSNR (32.91 dB), and best LPIPS (0.0554). Qualitatively, it preserved sharp edges and fine-grained textures (e.g., butterfly eyes, ballerina faces) that other methods blurred or distorted.
• Layering Quality: Unlike existing methods that produce fragmented or semantically entangled layers, AmodalSVG successfully decoupled objects. For example, it recovered a cat's tail and a dog's body hidden behind the cat, and reconstructed obscured background elements like bookshelves.
• Ablation Studies:
  • Removing the VLM or iterative peeling led to over-segmentation or missed objects.
  • Removing the hybrid inpainting resulted in empty background spaces.
  • Removing adaptive pruning led to excessive geometric complexity without perceptual gain, while removing addition failed to capture high-frequency details.
• Editing Capabilities: The framework demonstrated unique capabilities in object-level manipulation (e.g., moving a lighthouse, changing a giraffe's color, removing a shadow) which are impossible with modal vectorization methods.

5. Significance

AmodalSVG bridges the gap between the structural editability of vector graphics and the complexity of real-world natural images. By shifting the completion task to the raster domain using generative AI and then vectorizing the result, it overcomes the limitations of purely geometric tracing. This enables:

• Professional Design Workflows: Designers can now edit complex, occluded scenes in vector format as easily as they edit flat icons.
• Downstream Applications: The semantically organized layers facilitate applications in animation, game asset creation, and interactive media where object isolation and manipulation are critical.
• Advancement in Amodal Perception: It demonstrates a practical application of amodal perception (reasoning about occluded objects) in the context of image processing and computer graphics.

Limitations: The method relies on pre-trained VLMs, so errors in object identification can propagate. It also struggles with transparent objects (glass) and cyclic occlusions (e.g., a parrot gripping a branch that occludes the parrot), as these violate the strict global z-ordering of standard SVG formats. Future work aims to address these topological challenges.