💻 computer science

Abstraction in Style

This paper introduces Abstraction in Style (AiS), a generative framework that decouples structural abstraction from visual stylization to better capture the deep reinterpretation of geometry characteristic of illustrative and nonphotorealistic artistic styles.

Original authors: Min Lu, Yuanfeng He, Anthony Chen, Jianhuang He, Pu Wang, Daniel Cohen-Or, Hui Huang

Published 2026-04-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Min Lu, Yuanfeng He, Anthony Chen, Jianhuang He, Pu Wang, Daniel Cohen-Or, Hui Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to take a photograph of a real-life cat and turn it into a drawing that looks like it was painted by a famous cartoonist.

The Problem with Old Methods:
Most current AI tools act like a strict photocopy machine. They look at your photo and say, "Okay, I will paint over this cat with cartoon colors and brush strokes." But they keep the cat's exact shape, the exact curve of its ear, and the exact number of whiskers. If the cartoonist usually draws cats with round, squishy bodies and no whiskers, the old AI still draws a realistic cat with whiskers, just colored differently. It misses the spirit of the art style.

The New Solution: "Abstraction in Style" (AiS)
The authors of this paper created a new system called Abstraction in Style (AiS). Think of AiS not as a paintbrush, but as a two-step translator that understands both the shape of the object and the look of the art.

Here is how it works, using a simple analogy:

Step 1: The "Ghost Skeleton" (Structural Abstraction)

Before painting, the AI needs to understand how the artist thinks, not just how they paint.

The Input: You give the AI a photo of a real cat.
The "Ghost Skeleton": The AI first strips away all the fur, color, and texture, turning the cat into a simple, rough outline. But it doesn't stop there. It looks at the reference art (the cartoonist's drawings) and asks: "How does this artist simplify things?"
- If the artist turns round bodies into ovals and ignores small details, the AI re-draws the ghost skeleton to match that rule. It might turn the realistic cat's jagged ears into smooth triangles and merge its legs into a single block.
- The Result: This new, simplified shape is called the "Abstraction Proxy." It's no longer a photo; it's a "concept" of the cat, reshaped to fit the artist's logic.

Step 2: The "Painting Party" (Visual Stylization)

Now that the AI has the correct "concept" of the cat (the reshaped skeleton), it moves to the second stage.

The Transformation: The AI takes this reshaped skeleton and applies the specific colors, brush strokes, and textures from the reference art.
The Result: Because the skeleton was already reshaped to match the artist's style, the final painting looks perfectly coherent. It doesn't look like a photo with a filter; it looks like a genuine piece of art created by that specific artist.

Why This is a Big Deal

Think of it like cooking:

Old AI: Takes a raw steak, puts a "Italian Sauce" filter on it, and serves it. It still tastes like a tough, raw steak.
AiS: First, it cuts the steak into the shape of a pizza (because that's how the Italian chef serves meat), then it adds the sauce and cheese. The final dish actually tastes like a pizza.

The Secret Sauce: "Visual Analogy"

The paper uses a clever trick called Visual Analogy Transfer (VAT).
Imagine you show the AI a puzzle:

Top Row: "Here is a simple stick figure (A), and here is how the artist turns that stick figure into a cartoon (A')."
Bottom Row: "Here is a new stick figure (B). Can you use the same logic to turn B into a cartoon (B')?"

The AI learns the rule of the transformation (e.g., "make lines wiggly," "remove the nose") rather than just memorizing the picture. This allows it to handle new images it has never seen before.

The Takeaway

Abstraction in Style is a breakthrough because it separates changing the shape from changing the color.

Old way: Change color, keep shape.
New way: Change shape to match the artist's logic, then change the color.

This means you can now turn a realistic photo of a building into a sketch that looks like it was drawn by a child, or a complex machine into a simple icon, while keeping the image looking natural and consistent with the chosen art style. It gives the AI the freedom to "re-imagine" the world, not just repaint it.

1. Problem Statement

Conventional style transfer methods (e.g., Gatys et al., diffusion-based approaches) primarily focus on transferring surface-level visual traits such as texture, color, and stroke patterns while strictly preserving the input geometry. However, many artistic styles—particularly illustrative, sketch-based, and non-photorealistic styles—rely on structural abstraction. This involves the deliberate reinterpretation of structure, including:

Simplifying or smoothing away insignificant geometric features.
Introducing deliberate irregularities or breaking symmetry.
Distorting proportions to convey specific semantic or perceptual intent.

Existing methods struggle to capture this "deep abstraction" because they are designed to adhere to the original spatial structure. Consequently, they fail to generate coherent results for highly abstract styles where the geometry itself must be transformed, not just the appearance.

2. Methodology: Abstraction in Style (AiS)

The authors propose AiS, a generative framework that explicitly decouples structural abstraction from visual stylization. The process is formulated as a two-stage pipeline, both driven by a shared mechanism called Visual Analogy Transfer (VAT).

A. Core Concept: Two-Stage Pipeline

Stage 1: Structural Abstraction
- Goal: Reinterpret the target image's structure according to the abstraction logic of the reference style.
- Input: Target Image $\rightarrow$ Hidden Backbone $\rightarrow$ Abstraction Proxy.
- Hidden Backbone Construction: The target image is first converted into a "Hidden Backbone," a simplified structural representation. This is achieved via:
  - Vectorization: Converting the image into clean, flat-color vector paths to remove photographic noise.
  - Skeletonization & Erosion: Extracting a topological skeleton and applying morphological erosion to the filled shapes. The union of the skeleton and eroded residuals preserves connectivity and proportional volume without binding to precise contours.
- Transformation (A-VAT): A Visual Analogy Transfer model (A-VAT) learns the mapping from Hidden Backbone $\rightarrow$ Abstraction Proxy using reference pairs. It transforms the target's backbone into an intermediate proxy that reflects the reference's abstraction tendencies (e.g., smoothing, exaggerating) while maintaining subject identity.
Stage 2: Visual Stylization
- Goal: Render the abstracted proxy into a final image that matches the visual appearance of the reference style.
- Transformation (S-VAT): A second VAT model (S-VAT) learns the mapping from Abstraction Proxy $\rightarrow$ Final Stylized Output.
- Mechanism: This stage focuses purely on appearance (texture, color, stroke) applied to the already abstracted structure.

B. The Engine: Visual Analogy Transfer (VAT)

Both stages utilize VAT, a mechanism based on the visual analogy formulation: $A \rightarrow A' :: B \rightarrow B'$ .

Input Format: A $2 \times 2$ grid image where the top row shows a reference pair ( $A \rightarrow A'$ ) and the bottom row shows the input ( $B$ ) and a masked target ( $B'$ ).
Implementation:
- Built upon a pre-trained Diffusion Transformer (DiT) (specifically FLUX.1-Fill-dev).
- Uses LoRA (Low-Rank Adaptation) to fine-tune a lightweight adapter on a small set of exemplars (5–20 pairs per style).
- Operates entirely in image space, learning complex transformations without explicit geometric supervision or handcrafted rules.

3. Key Contributions

Explicit Decoupling: The paper introduces the first framework to explicitly separate structural abstraction from visual stylization, treating abstraction as a learnable, transferable process rather than an implicit byproduct.
Hidden Backbone Representation: A novel intermediate representation (skeleton + eroded regions) that strips away appearance while preserving topological structure and proportional volume, serving as a flexible scaffold for geometric reimagination.
Visual Analogy Transfer (VAT): A general, reusable image-space mechanism that learns style-specific transformations from small exemplar sets (averagely 10 images) without requiring paired training data or geometric labels.
Controllability: By separating A-VAT and S-VAT, the framework allows for "mix-and-match" generation. Users can combine different abstraction logics with different rendering styles to create hybrid designs.

4. Results and Evaluation

Qualitative Performance: AiS successfully generates coherent stylized images across diverse illustrative styles (e.g., sketches, cartoons) where the geometry is significantly altered. It outperforms state-of-the-art baselines (StyleID, StyleAlign, Nano Banana Pro, etc.) which tend to preserve the original input geometry, resulting in less coherent abstract representations.
Quantitative Metrics:
- Style Similarity (CSD): AiS achieved the highest score (0.72) compared to baselines (0.58–0.64).
- Perceptual Similarity (LPIPS): AiS achieved the lowest score (0.47), indicating better alignment with the reference style's visual characteristics.
User Study: In a blind evaluation with 20 test cases, AiS was preferred by 50% of users, significantly outperforming the next best method (Nano Banana at 35%).
Ablation Studies:
- Removing the eroded region cues in the backbone led to geometric degradation.
- Removing the two-stage design (using a single-stage AS-VAT) resulted in outputs that adhered too rigidly to input geometry, failing to capture the abstraction logic.
- Removing the abstraction stage entirely failed to generalize the visual logic of the reference.

5. Significance

The significance of AiS lies in its paradigm shift from "appearance transfer" to "structural reinterpretation."

Bridging Art and AI: It addresses a critical gap in computer graphics where AI struggles to replicate the high-level reasoning artists use to simplify and distort forms.
Scalability: The method works effectively with very small datasets (few-shot learning), making it accessible for niche artistic styles.
Future Direction: It establishes a foundation for future work in "controlled semantic warping," suggesting that separating abstraction from appearance is essential for achieving expressive, non-photorealistic generation.

In summary, Abstraction in Style provides a robust solution for generating stylized images that respect the logic of a style, not just its look, enabling AI to produce truly abstract and illustrative art rather than just textured photographs.