🤖 AI

CSGen: A Multi-Domain Curvilinear Structure Generation Model via Hierarchical Multimodal Diffusion

The paper introduces CSGen, a hierarchical multimodal diffusion model that leverages a large-scale multi-domain dataset, a progressive control strategy, and a sparsity-aware loss mechanism to achieve high-fidelity, controllable generation of images with precise curvilinear structures across diverse multimedia applications.

Original authors: Zhe Shan, Ziming Yang, Lei Zhou, Wenwen Zhang, Cong Lin, Xia Xie

Published 2026-08-06

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhe Shan, Ziming Yang, Lei Zhou, Wenwen Zhang, Cong Lin, Xia Xie

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot artist to draw the world. You have shown it millions of pictures of cats, cars, and sunsets, so it knows how to paint a fluffy dog or a shiny car perfectly. But then you ask it to draw something very different: a single, tiny, winding crack in a sidewalk, or a delicate network of blood vessels inside an eye. Suddenly, the robot gets confused. It tries to paint a whole sidewalk or a whole eye, drowning out the tiny crack you wanted. This is the problem of "curvilinear structures"—long, thin, winding lines that are crucial for things like medical scans, road maps, and checking if bridges are safe. These lines are so thin and sparse that they get lost in the noise of the background.

To solve this, scientists use "diffusion models." Think of these models like a sculptor who starts with a block of noisy, static-filled clay and slowly chips away the noise to reveal a clear image. Usually, this works great for big, chunky objects. But when the object is a fragile, hair-thin line, the sculptor often smudges it away or breaks it into pieces because the line is so small compared to the rest of the picture. The big question is: how do we teach this digital sculptor to be incredibly gentle and precise with these tiny, fragile lines without losing the big picture?

Enter CSGen, a new model designed specifically to master these tricky, winding lines. The researchers behind this project realized that the usual way of training these AI artists wasn't working for thin structures. They built a brand-new training system that acts like a strict but helpful art teacher. First, they gathered a massive library of over 24,000 examples of these winding lines from five different worlds: the veins in your eye, the arteries in your heart, cracks in concrete, veins in leaves, and roads on a map. This gave the AI a huge variety of "thin line" patterns to learn from.

But just having the pictures wasn't enough. The researchers invented two clever tricks to help the AI focus. The first trick is like a "step-by-step" lesson plan. Instead of asking the AI to draw the whole complex scene at once, they first teach it the basic shape and connection of the lines (the skeleton), and only later add the details like color and texture. This stops the AI from getting confused and breaking the lines apart. The second trick is a special "spotlight" for the training process. Since the lines are so thin, the AI usually ignores them. The researchers programmed the model to pay extra attention to the thinnest, most fragile parts of the drawing, essentially telling it, "Don't skip these tiny bits; they are the most important!"

The results show that this new approach works. When they tested CSGen, it created images where the winding lines were much more accurate and connected than previous methods. It didn't just look better; when other computer programs tried to use these generated images to learn how to find cracks or blood vessels, those programs got better at their jobs too. The paper suggests that by combining a huge, diverse dataset with these smart training steps, we can finally get AI to handle the delicate, winding structures that are so important for keeping our roads safe and our medical diagnoses accurate.

Technical Summary: CSGen – A Multi-Domain Curvilinear Structure Generation Model via Hierarchical Multimodal Diffusion

1. Problem Statement

Curvilinear structure analysis (e.g., retinal vessels, road networks, cracks) is a fundamental task in multimedia, yet the controllable generation of high-fidelity images featuring these structures remains an under-explored challenge. Unlike common objects, curvilinear structures are characterized by:

Extreme Spatial Sparsity: They occupy a tiny fraction of image pixels, creating severe foreground-background signal imbalances.
Topological Rigidity: They require precise geometric connectivity and continuity, which standard diffusion models often fail to maintain.
Data Scarcity: High-quality, pixel-level annotated datasets for these specific structures are difficult to construct due to the cost of manual labeling and the lack of identifiable instances in natural scenes.

Existing diffusion models (e.g., SD3.5) struggle with these tasks because their pre-trained priors are biased toward dense, object-centric scenes. When applied to curvilinear structures, they suffer from semantic drift (failing to align text prompts with fine-grained geometry) and topological fragmentation (breaking thin, fragile branches).

2. Methodology: CSGen

The authors propose CSGen, a hierarchical multimodal diffusion model built upon the Stable Diffusion 3.5 (SD3.5) backbone. The framework integrates three core innovations to address the specific challenges of curvilinear generation:

A. Multi-Domain, Multimodal Dataset Construction

To overcome data scarcity, the authors constructed a comprehensive dataset comprising 24,678 samples across 5 domains (Retinal Blood Vessels, Coronary Arteries, Concrete Cracks, Leaf Veins, and Road Networks) with 7 types of annotations.

Structural Control Maps: Instead of relying solely on masks, the system generates four complementary control maps: Width Maps (distance-based thickness), Sketches (skeleton-edge fusion), Edge Maps (Canny detection), and Partial Sketches (random erasing for robustness).
Hierarchical Semantic Descriptors: Text prompts are constructed using a two-level approach:
1. Rule-Based Descriptors: Discretized geometric attributes (e.g., tortuosity, width) to provide stable structural anchors.
2. VLM-Based Descriptors: Detailed environmental descriptions generated by a Vision-Language Model, constrained by metadata to prevent hallucinations.

B. Hierarchical Progressive Control Strategy (HPCS)

To mitigate semantic drift and ensure topological integrity, HPCS decouples geometric topology from visual context through a phased signal injection:

Geometric Anchoring ( $h_{geo}$ ): Initially, the model is guided by stable geometric attributes encoded via CLIP (from rule-based descriptors). This establishes the core topology.
Contextual Refinement ( $h_{ctx}$ ): As training progresses, fine-grained scene details encoded via T5 (from VLM descriptors) are gradually injected.
Mechanism: A curriculum schedule controls the injection weight $\lambda(\tau)$ , ensuring the model learns the structural skeleton before refining environmental textures.

C. Sparsity-Aware Loss Re-weighting Mechanism (SLRM)

Standard loss functions (e.g., MSE) treat all spatial coordinates equally, causing the model to ignore sparse, thin structures. SLRM addresses this by:

Quantifying Difficulty: Calculating the equivalent diameter ( $d_i$ ) for each sample's structure.
Dynamic Re-weighting: Assigning higher optimization weights to samples with smaller diameters (thinner, more fragile structures) using a logarithmic function. This forces the model to prioritize the preservation of topologically critical, low-contrast regions during flow-matching optimization.

3. Key Contributions

CSGen Model: A novel hierarchical multimodal diffusion model capable of synthesizing high-fidelity images aligned with multiple control conditions (layout maps and text prompts) specifically for curvilinear structures.
Comprehensive Dataset: The creation of a 24K+ sample, multi-domain dataset with diverse annotations, providing a robust prior for training unified generation models in specialized domains.
HPCS & SLRM: The introduction of a progressive control strategy to decouple topology from context and a sparsity-aware loss mechanism to enhance attention on fragile structures, significantly improving boundary precision and topological consistency.

4. Experimental Results

Extensive experiments demonstrate that CSGen outperforms state-of-the-art (SOTA) methods, including FLUX, Qwen, and fine-tuned SD3.5 variants (LoRA, ControlNet).

Generation Quality: CSGen achieves superior performance in image-level metrics (FID: 98.5, IS: 4.19) and semantic alignment (VQA: 0.96).
Structural Accuracy: In parser-based evaluations (using SegFormer to segment generated images), CSGen significantly improves topology-aware metrics. Notably, the clDice (a topology-consistency metric) improved from 0.628 (baseline) to 0.766 (full model).
Ablation Studies:
- The SLRM provided the most significant boost, particularly in preserving thin branches.
- HPCS effectively stabilized the flow matching optimization, transitioning from coarse anchors to rich details.
Downstream Utility: When used as synthetic data for training segmentation models (DeepLabV3+, SegFormer) on blood vessels, cracks, and road networks, CSGen-generated data consistently improved performance (e.g., +2.64 mIoU on Blood vessels), whereas standard ControlNet augmentation sometimes degraded performance.

5. Significance

The paper positions CSGen as a scalable, data-centric paradigm for analyzing complex curvilinear structures. It claims to bridge a critical gap in multimedia analysis by shifting the generation process from stochastic approximation to topologically consistent reconstruction. By addressing the specific challenges of spatial sparsity and topological connectivity, CSGen enables the generation of high-quality, controllable data that can serve as a robust foundation for downstream tasks such as data augmentation, structural health monitoring, and medical image analysis. The authors emphasize that their approach allows for the generation of images with precise structural adherence even for the most sparse and delicate structures, a capability lacking in current general-purpose diffusion models.