DesigNet: Learning to Draw Vector Graphics as Designers Do

1. Problem Statement

While AI has advanced significantly in raster image generation, generating Scalable Vector Graphics (SVG) remains challenging. Existing methods often fail to meet professional design standards due to two main issues:

• Geometric Inconsistency: Neural networks often produce vector outlines with poor continuity (jagged transitions) and misaligned axes (slanted lines that should be horizontal/vertical), making them difficult for human designers to refine.
• Discretization Artifacts: Prior state-of-the-art (SOTA) models often rely on discretized coordinate tokens to enforce alignment, which sacrifices geometric precision and prevents smooth interpolation.
• Workflow Mismatch: Current outputs are often "uneditable" or structurally messy, failing to integrate seamlessly into professional tools like Adobe Illustrator or FontForge.

The core goal is to bridge the gap between neural generation and human design practice by equipping models with explicit controls for continuity (smoothness at junctions) and alignment (horizontal/vertical snapping).

2. Methodology: DesigNet

DesigNet is a hierarchical Transformer-VAE that operates natively on continuous SVG command sequences. It introduces two novel differentiable modules to enforce geometric constraints.

A. Data Representation

• Continuous Parameterization: Unlike prior works that discretize coordinates, DesigNet uses continuous coordinates for all command arguments. This avoids quantization loss and enables smooth latent space interpolation.
• Command Format: Glyphs are represented as sequences of paths, where each path is a sequence of commands (MoveTo, LineFromTo, CurveFromTo). Arguments are represented as 4-point tuples $(p_1, p_2, p_3, p_4)$, unifying lines and cubic Bézier curves.

B. Architecture: Hierarchical Partitioned VAE

The model employs a two-level hierarchy to balance global style with local geometric detail:

1. Path-Level Encoding/Decoding: A Transformer processes individual paths independently to capture local geometric structures.
2. Glyph-Level Encoding/Decoding: A second Transformer aggregates path embeddings to capture the global font style (weight, slant, contrast).
3. Partitioned Latent Space: Instead of compressing a glyph into a single vector, DesigNet uses:
   • A global latent ($z$) for overall style.
   • Path-level latents ($z_j$) for fine-grained contour details.
     This prevents the "bottleneck" effect where fine details are lost during compression.

C. Key Modules: Self-Refinement

The core innovation lies in two differentiable modules that predict and enforce geometric rules after the initial decoding but before the final output. These use Straight-Through Estimators (STE) to allow gradients to flow through discrete decisions during training.

1. Continuity Self-Refinement Module:

   • Prediction: Predicts the continuity level ($C_0$, $G_1$, or $C_1$) for every junction between curve segments.
     • $C_0$: Geometric continuity (shared endpoint).
     • $G_1$: Collinear tangents (smooth visual transition).
     • $C_1$: Collinear tangents with equal magnitude (mathematically smooth).
   • Refinement: A deterministic operator adjusts Bézier control points to satisfy the predicted continuity level. For example, if $G_1$ is predicted, control points are moved to ensure tangents are collinear.

2. Alignment Self-Refinement Module:

   • Prediction: Predicts whether a line segment should be Horizontal ($H$), Vertical ($V$), or unaligned ($\emptyset$).
   • Refinement: A "snapping" operator adjusts the endpoints of the line to the nearest horizontal or vertical axis based on the prediction.

D. Training Objective

The loss function combines standard VAE objectives with specific geometric supervision:

• Reconstruction Loss ($L_{rec}$): Command type classification, argument regression, visibility, and endpoint consistency.
• KL Divergence ($L_{KL}$): Regularizes the latent space.
• Continuity Loss ($L_{cont}$): Cross-entropy loss with a cost-sensitive matrix (penalizing $C_0 \leftrightarrow C_1$ errors more heavily).
• Alignment Loss ($L_{align}$): Cross-entropy loss for axis alignment.
• Coupling: Because the refinement modules are differentiable (via STE), errors in predicting continuity or alignment directly impact the reconstruction error, forcing the model to learn accurate geometric distributions.

3. Key Contributions

1. Differentiable Self-Refinement Modules: Novel modules that explicitly predict and enforce $C_0/G_1/C_1$ continuity and axis alignment, making outputs structurally clean and editable.
2. Continuous Parameterization: Replacing discrete tokens with continuous coordinates, eliminating quantization artifacts and enabling smooth interpolation between fonts.
3. Partitioned Latent Space: A hierarchical VAE design that separates global style from local path geometry, preserving fine-grained details while maintaining stylistic coherence.
4. Font-Level Generation: Extending the model to generate entire alphabets from a few reference glyphs (one-shot/few-shot learning) with high fidelity.

4. Experimental Results

The authors evaluated DesigNet on Latin fonts, Chinese fonts, and Icons.

• Reconstruction Quality: DesigNet achieves competitive or superior performance in Intersection-over-Union (IoU), $\ell_1$ distance, and Reconstruction Error (Chamfer distance) compared to SOTA models like DeepSVG and DeepVecFont-v2.
• Geometric Accuracy:
  • Continuity: DesigNet significantly outperforms baselines in continuity accuracy (e.g., 0.886 vs. 0.567 for DeepSVG in ablation studies).
  • Alignment: The self-refinement module boosts alignment accuracy from ~0.37 (without refinement) to 0.969 in the ablation study.
• Qualitative Improvements: Visualizations show smoother curve transitions, perfectly aligned stems/crossbars, and cleaner structures that load easily into professional software (FontForge, Illustrator).
• Generalization: The model successfully generalizes to Chinese characters (complex structures) and icons, demonstrating robustness beyond Latin typography.
• Interpolation: The continuous latent space allows for smooth, meaningful interpolation between different font styles (e.g., weight and slant).

5. Significance

• Bridging AI and Design: DesigNet moves beyond "generating images" to "generating design assets." By mimicking designer workflows (snapping, continuity control), the outputs are immediately usable by human professionals.
• Overcoming Discretization: It proves that continuous coordinate representations, when paired with geometric refinement modules, can outperform discretized approaches in both precision and editability.
• Practical Application: The generated SVGs are structurally sound, reducing the manual cleanup time required for AI-generated vector graphics, which is a major bottleneck in current creative AI pipelines.
• Future Direction: The paper highlights that while geometric consistency is solved, future work should focus on compositional generation (reusing motifs like serifs) to achieve perfect stylistic consistency across complex scripts like Chinese.