💻 computer science

Controlling Your Image via Simplified Vector Graphics

This paper introduces a novel image synthesis framework that enables precise, element-level control over image generation by parsing images into hierarchical vector graphics representations and leveraging them to guide the creation of photorealistic outputs with intuitive modifications to shapes, colors, and objects.

Original authors: Lanqing Guo, Xi Liu, Yufei Wang, Zhihao Li, Siyu Huang

Published 2026-02-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lanqing Guo, Xi Liu, Yufei Wang, Zhihao Li, Siyu Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to edit a photo using a standard AI image generator. You tell it, "Make the cat blue," but the AI might change the whole background, or it might make the cat look like a blue blob instead of a cat. It's like trying to fix a specific ingredient in a soup by shouting at the whole pot; you often end up ruining the whole dish.

Vec2Pix is a new tool that solves this by changing the "language" the AI speaks. Instead of talking to the AI in pixels (tiny colored dots) or vague text prompts, Vec2Pix talks to it using Vector Graphics (SVGs).

Here is how it works, broken down into simple analogies:

1. The Magic of "Layers" (The Cake Analogy)

Think of a complex image not as a single flat painting, but as a multi-layered cake.

Layer 1: The background (the table).
Layer 2: The middle ground (the cake stand).
Layer 3: The foreground (the cake itself).
Layer 4: The details (the cherry on top).

Traditional AI sees the whole cake as one big mess of frosting. Vec2Pix sees the layers. It can tell you, "I only want to change the cherry," without touching the cake or the table. This is called Layer-wise Control. You can move the cherry to a different layer, change its color, or even swap the cherry for a strawberry, and the AI knows exactly where to put it without messing up the rest of the scene.

2. The "Blueprint" vs. The "Painting" (The Vector Analogy)

Most images are like finger paintings—once you put a dot of paint down, it's hard to move it without smearing the whole thing.
Vector Graphics (SVGs) are like architectural blueprints or cut-out paper shapes.

A blueprint defines a shape by its lines and curves, not by the color of the paper.
If you want to change a square into a circle in a blueprint, you just redraw the line. The rest of the building stays perfect.

Vec2Pix takes a real photo and instantly turns it into this "blueprint" format. Because the AI understands the blueprint, you can edit the "lines" (the shape of a mountain, the color of a car) and the AI knows exactly how to redraw the photo to match those new lines.

3. The "Smart Translator" (The Noise Prediction)

Here is the tricky part: How do you turn a simple line drawing (SVG) back into a photorealistic photo?
Imagine you have a rough sketch of a face. You want the AI to turn it into a real photo, but you don't want it to guess the wrong nose or eyes.

Vec2Pix uses a Smart Translator (called the NPV module).

Old way: The AI guesses the starting point of the photo from a random noise cloud. It's like trying to find a needle in a haystack.
Vec2Pix way: The Smart Translator looks at your SVG blueprint and says, "Okay, based on these lines, the photo should start with this specific pattern of noise." It guides the AI right from the very first step, ensuring the final photo matches your blueprint perfectly.

4. What Can You Actually Do?

With Vec2Pix, editing becomes as easy as using a graphic design program, but with the power of a super-smart AI.

Swap Objects: You can tell the AI, "Replace the dog with a cat," and it will keep the lighting and shadows perfect.
Move Things: You can drag a building from the background to the foreground, and the AI will figure out how to make it look 3D and realistic.
Fix Mistakes: If an AI generates a hand with six fingers, you can draw a simple SVG outline of a five-finger hand, and Vec2Pix will "heal" the image to fix the fingers instantly.

The Big Picture

Before this, editing AI images was like trying to sculpt a statue by throwing clay at it and hoping it lands in the right shape. Vec2Pix gives you a chisel and a mold. It lets you control the image element-by-element, ensuring that when you change the color of a shirt, the rest of the person stays exactly the same.

It bridges the gap between creative freedom (doing whatever you want) and precision (getting exactly what you asked for).

1. Problem Statement

Recent advances in image generation (e.g., Diffusion models, Flow models) have achieved high visual quality but suffer from a fundamental limitation: lack of element-level controllability.

Current Limitations: Existing methods rely on coarse controls (layouts, sketches) or localized interactions (drag-and-drop). These approaches often fail to support intuitive, fine-grained modifications such as changing an object's shape, altering specific colors, or adding/removing objects without disrupting background consistency or introducing artifacts.
The Gap: There is a need for a representation that bridges high-quality synthesis with practical usability, allowing users to manipulate individual semantic elements (geometry, color, structure) intuitively while maintaining photorealism.

2. Methodology: Vec2Pix Framework

The authors propose Vec2Pix, a framework that uses Simplified Vector Graphics (VGs) as a hierarchical, semantically aligned representation to guide image generation. The system operates in a closed loop: Image $\leftrightarrow$ SVG.

A. Efficient and Simplified Vector Graphics (Image-to-SVG)

Unlike traditional vectorization methods that produce complex, fragmented paths, Vec2Pix creates a hierarchical and semantically aligned SVG representation.

Hierarchical Initialization: The system uses foundation models (Segment Anything Model - SAM and Diffusion priors) to decompose an image into semantic layers (e.g., background, mid-ground, foreground).
Semantic Alignment: Masks are generated for different simplification levels. Redundant masks are discarded, and each layer is associated with a parent mask to form a hierarchy (e.g., a "Robot" layer containing "Head," "Body," "Legs").
Polygonization & Simplification: Semantic masks are converted into editable SVG paths using the Douglas–Peucker algorithm. Complex polygons are split and fitted with a limited number of cubic Bézier curves (max 8 segments per side) to ensure editability.
Differentiable Rasterization: To optimize the vector graphics against the original image, the authors use Bézier Splatting. Crucially, they enforce a semantic-aligned rendering formulation where opacity is fixed at 1.0. This prevents alpha-blending from corrupting the exclusivity of semantic regions, ensuring the vector structure remains coherent during optimization.
Efficiency: This pipeline achieves a 7 $\times$ speedup compared to previous semantic-layered vectorization methods (like LIVSS) while maintaining high reconstruction quality.

B. Vector-Guided Controllable Image Generation (SVG-to-Image)

The generation module translates the SVG and text prompts into photorealistic images using a Flow Matching model (based on FLUX.1-dev).

Two-Stage Training Strategy:
1. Stage 1 (Adaptation): LoRA adapters are integrated into the transformer blocks of the base model. The SVG features are encoded and concatenated with noise and text embeddings. A Multimodal Attention (MMA) mechanism fuses visual (SVG) and textual cues.
2. Stage 2 (Noise Prediction from Vectors - NPV): Instead of sampling initial noise from a standard Gaussian distribution, a dedicated NPV module predicts the mean ( $\mu$ ) and variance ( $\sigma^2$ ) of the initial latent noise conditioned on the rendered SVG.
  - This allows the model to inject structural and semantic information directly at the start of the generation process.
  - The training objective includes Flow Matching loss, KL divergence loss, and a Covariance Loss to encourage spatial independence across latent channels.
Result: This approach ensures precise alignment between the SVG geometry and the generated image, allowing for high-fidelity reconstruction and editing.

3. Key Contributions

Simplified Vector Graphics Representation: A novel hierarchical vector parsing method that is semantically aligned and structurally coherent, offering an interpretable and manipulable representation for user interaction.
Vector-Guided Noise Prediction (NPV): A mechanism where a tunable encoder derives the spatially variant initial noise of diffusion models directly from vector graphics, significantly improving structural and semantic alignment.
Element-Wise Controllable Generation: A framework that supports flexible operations including layer-wise generation, object-level editing (shape, color, position), and composition of real images with SVGs.

4. Experimental Results

The authors evaluated Vec2Pix against state-of-the-art baselines (Canny edges, Depth maps, Strokes, Segmentation Masks) and other editing tools (GPT-4o, Qwen-Image, ICEdit).

Quantitative Performance:
- Reconstruction: Vec2Pix (with NPV) achieved the best scores across all metrics: FID (15.52), LPIPS (0.3515), SSIM (0.49), and PSNR (17.77).
- Editing: In editing tasks, Vec2Pix significantly outperformed other representations, achieving an FID of 17.84 and PSNR of 17.72, demonstrating superior ability to preserve context while modifying specific elements.
Qualitative Capabilities:
- Layer-wise Generation: Users can insert objects into specific depth layers (e.g., placing a building behind trees).
- Object Editing: Precise modification of shape (e.g., changing a dog to a cat), color, and size.
- Composition: Seamless blending of real images and vector graphics.
- Artifact Removal: The system can correct generation flaws (e.g., distorted fingers or shoes) by editing the SVG and re-generating.
Ablation Studies:
- Removing the NPV module resulted in lower alignment between the SVG and the generated image.
- Adjusting the vector scale allowed flexible adaptation to complex effects (reflections, smoke) without being overly constrained by geometry.

5. Significance

Vec2Pix establishes a new paradigm for controllable image generation. By moving away from pixel-space conditioning (like sketches or masks) to hierarchical vector graphics, it solves the trade-off between high-fidelity synthesis and granular editability.

Practical Impact: It enables intuitive workflows for designers and creators, allowing for "what-you-see-is-what-you-get" editing where structural changes in vectors translate immediately to realistic visual changes.
Technical Advancement: The integration of Bézier Splatting for fast vectorization and Noise Prediction from Vectors for guided generation sets a new standard for how structural priors can be leveraged in diffusion/flow models.

In summary, Vec2Pix bridges the gap between the precision of vector graphics and the photorealism of generative AI, offering a robust solution for element-level image manipulation.