DRG-Font: Dynamic Reference-Guided Few-shot Font Generation via Contrastive Style-Content Disentanglement

1. Problem Statement

Few-shot Font Generation (FFG) aims to generate a complete set of character glyphs in a target font style using only a few reference images. While deep generative models (GANs, Diffusion) have advanced this field, existing methods face three primary limitations:

• Loss of Local Characteristics: Methods often struggle to retain discernible local details and fine-grained stroke structures in generated samples.
• Script Dependency: Many approaches rely on predefined stroke decompositions (e.g., splitting characters into strokes or radicals), which limits their applicability to scripts with well-defined stroke rules (like Chinese) and fails to generalize to scripts without such rules (like Latin).
• Suboptimal Reference Selection: Randomly selecting style references from a pool often leads to poor style adaptation, as the structural similarity between the reference and the target content is ignored.
• Computational Overhead: Diffusion-based methods, while high-quality, are computationally expensive and less practical for many deployment scenarios.

2. Methodology

The proposed DRG-Font framework addresses these issues through a dynamic, contrastive learning approach that disentangles style and content embeddings without relying on rigid stroke decomposition.

A. Reference Selection (RS) Module

Before generation, the system dynamically selects the optimal style reference ($x_s$) from a pool of candidates for a given target character ($x_c$).

• Mechanism: It uses a Stroke Matching Comparator (SMC) to measure structural similarity.
• Process:
  1. Skeletonization is performed on the candidate and target characters.
  2. "Salient Points" (nodes with degree 1 or >2) are identified to decompose the skeleton into strokes.
  3. Strokes are described by three features: normalized length, average curvature, and orientation distribution.
  4. A pairwise cosine similarity is calculated between stroke descriptors. The candidate with the highest similarity score is selected as the style reference.

B. Generator Network ($G$)

The generator consists of an Encoder ($E_{sc}$) and a Decoder ($D_{sc}$), designed to learn disentangled style and content spaces.

• Style-Content Encoder ($E_{sc}$):

  • Uses deformable convolutions to capture geometric invariance.
  • Processes the input through four downsampling blocks.
  • Multi-scale Style Head Block (MSHB): Extracts style features from three latent scales using statistical projections (mean and variance) to create style embeddings.
  • Multi-scale Content Head Block (MCHB): Extracts content (shape) features using convolution and pooling to create content embeddings.
  • Contrastive Learning: The network is trained to separate style and content embeddings using a contrastive strategy, ensuring that style embeddings are invariant to content changes and vice versa.

• Style-Content Decoder ($D_{sc}$):

  • Takes the style embedding from the selected reference ($e_{style}^s$) and the content embedding from the target character ($e_{content}^c$).
  • Multi-Fusion Upsampling Block (MFUB): Progressively upsamples the content latent while injecting multiscale style features via AdaIN (Adaptive Instance Normalization).
  • Gating Mechanism: A style-conditioned gating mechanism modulates the features during upsampling to ensure precise style adaptation.

C. Discriminator Network ($D$)

A multi-task PatchGAN discriminator provides adversarial guidance and auxiliary supervision:

• Adversarial Head: Distinguishes real vs. generated glyphs.
• Classification Heads: Predicts style and content classes to enforce consistency in the generated output.

D. Training Objective

The model is optimized using a hybrid loss function comprising six components:

1. Reconstruction Loss ($L_{recon}$): Pixel-wise $\ell_1$ distance.
2. Perceptual Loss ($L_{perc}$): VGG-based feature similarity.
3. Adversarial Loss ($L_{adv}$): Hinge loss for realism.
4. Classification Loss ($L_{cls}$): Cross-entropy for style/content classification.
5. Disentanglement Loss ($L_{dist}$): Circle Loss applied to style and content embeddings to ensure compact clusters for positive pairs and large margins for negative pairs.
6. Latent Reconstruction Loss ($L_{latent}$): Uses a frozen Stable Diffusion v2 VAE Encoder to align generated and ground-truth images in the latent space, ensuring high-level structural consistency beyond pixel-level supervision.

3. Key Contributions

1. Dynamic Reference Selection: Introduces an RS Module that selects the structurally closest style reference, significantly improving the retention of glyph characteristics.
2. Contrastive Disentanglement: Proposes a novel strategy to decompose style and content into separate embedding spaces using contrastive learning (Circle Loss), avoiding the need for script-specific stroke decomposition.
3. Hybrid Architecture: Combines a multi-scale encoder/decoder with a latent reconstruction mechanism (Stable Diffusion VAE) and adversarial guidance to achieve high-fidelity generation.
4. Generalization: The method generalizes effectively across different scripts (Latin and Chinese) without script-specific architectural changes.

4. Experimental Results

The method was evaluated on English (811 fonts) and Chinese (521 fonts) datasets, comparing against SOTA methods like FANNET, MA-Font, PatchFont, FASTER, and DA-Font.

• Quantitative Performance: DRG-Font achieved state-of-the-art results across all metrics (L1, RMSE, SSIM, LPIPS) for both Seen and Unseen fonts.
  • Example: On Unseen English, it achieved an L1 of 0.072 and SSIM of 0.739, outperforming the next best (DA-Font: 0.074, 0.713).
• User Study: In a subjective evaluation with 76 participants, the proposed method was preferred in 53.42% of cases for English and 55.66% for Chinese, significantly outperforming all competitors.
• Ablation Studies:
  • Removing the RS Module caused a significant drop in quality (e.g., L1 increased by ~26%).
  • Removing $L_{cls}$ and $L_{latent}$ resulted in lower reconstruction fidelity.
  • The optimal embedding dimension for each head was found to be 256; deviations (128 or 512) degraded performance.

5. Significance

DRG-Font represents a significant advancement in few-shot font generation by shifting away from rigid, script-dependent structural decomposition toward a dynamic, data-driven approach.

• Generalizability: By not relying on predefined stroke rules, it can handle diverse scripts (Latin, Chinese, and potentially others) with a unified architecture.
• Quality vs. Efficiency: It achieves diffusion-like quality in terms of sharpness and structural integrity but operates with the efficiency of a GAN-based architecture, making it more practical for real-world applications.
• Robustness: The dynamic selection of references ensures that the generated glyphs maintain high structural consistency even when the available style references are limited or varied.