Technical Summary: InkDiffuser

Problem Statement

Current methods for generating Chinese calligraphy, particularly in few-shot and one-shot settings, suffer from significant limitations in visual fidelity and artistic authenticity. While existing approaches (both GAN-based and diffusion-based) can synthesize structurally plausible glyphs, they often fail to reproduce the complex ink morphology characteristic of real brush writing. Specifically, these methods tend to produce outputs that appear overly digital, flat, or blurred, lacking the intricate ink diffusion, stroke edge variations, and natural interaction between ink and paper found in genuine calligraphy. Furthermore, standard content encoders in diffusion models often prioritize low-frequency features, leading to the loss of high-frequency structural details such as sharp corners and stroke edges, which is particularly detrimental for complex Chinese characters.

Methodology

The authors propose InkDiffuser, a diffusion-based image-to-image framework designed for one-shot Chinese calligraphy synthesis. The framework is built upon the Denoising Diffusion Probabilistic Model (DDPM) and introduces two core mechanisms to address the aforementioned limitations:

1. Spatio-Temporal Adaptive Fusion (STAF) Module

To counter the loss of high-frequency structural details, InkDiffuser employs a dual-encoder architecture for the content stream:

• Dual Encoding: A main encoder ($E_c$) processes the standard content image, while a dedicated high-frequency encoder ($E_{hf}$) processes a high-pass filtered (Sobel edge) version of the content image.
• Residual Fusion: The STAF module fuses these features via a residual framework. It learns to add a hierarchically-modulated detail increment to the content backbone.
• Adaptive Control: The fusion is governed by a three-dimensional control mechanism:
  • Spatial: A pixel-wise attention map ensures high-frequency details are injected only into salient regions (e.g., stroke edges) while suppressing noise in smooth areas.
  • Layer-wise: Weights are assigned to favor shallow, high-resolution layers for fine-grained detail injection while preserving semantic integrity in deeper layers.
  • Temporal: The fusion intensity is modulated by the diffusion timestep, providing strong structural guidance during early, high-noise stages and tapering off as the image clarifies to prevent artifacts.

2. Differentiable Ink Structure (DIS) Loss

To explicitly regularize ink morphology and enable end-to-end optimization of brush-like features, the authors propose the DIS loss. This loss decomposes the ink generation problem into three objectives using soft, differentiable morphological operations:

• Differentiable Morphology: Traditional morphological operations (erosion/dilation) rely on non-differentiable min/max functions. InkDiffuser approximates these using sigmoid functions scaled by a temperature parameter $\tau$, allowing gradients to flow through the morphological analysis.
• Three-Component Loss:
  1. Core Structural Consistency ($L_c$): Uses soft erosion to isolate stroke cores, ensuring the skeletal structure remains accurate even under diffusion effects.
  2. Diffusion Boundary Consistency ($L_b$): Uses the difference between soft dilation and erosion to create a boundary mask, focusing supervision on the ink-spread regions to control the extent of diffusion.
  3. Diffusion Smoothness ($L_{lap}$): Employs a Laplacian operator to minimize the difference in edge responses between generated and target images, encouraging the smooth, rounded transitions typical of real ink.
• Circular Structuring Elements: To preserve the natural smoothness of calligraphic curves, circular kernels are used instead of traditional square ones.

The total training objective combines the standard diffusion MSE loss, a perceptual loss (VGG-based), and the weighted DIS loss.

Key Contributions

1. InkDiffuser Framework: A novel one-shot generation framework that effectively balances the preservation of intricate structural integrity with the rendering of realistic ink diffusion dynamics.
2. Spatio-Temporal Adaptive Fusion (STAF): A module that extracts and fuses high-frequency components from content images to strengthen stroke representation, significantly improving the synthesis of complex characters.
3. Differentiable Ink Structure (DIS) Loss: A tailored loss function that incorporates soft morphological operators to provide explicit supervision on ink trace decomposition (core vs. boundary), enabling the model to learn realistic ink spread and boundary control in an end-to-end manner.

Experimental Results

The authors evaluated InkDiffuser on a dataset of 293 Chinese fonts, split into seen and unseen sets, using two evaluation benchmarks: UFUC (Unseen Fonts, Unseen Characters) and UFSC (Unseen Fonts, Seen Characters).

• Quantitative Performance: InkDiffuser achieved state-of-the-art performance across all five metrics (LPIPS, SSIM, L1, RMSE, PSNR) on both benchmarks. Notably, on the challenging UFUC setting, it improved SSIM by 0.0242 and reduced L1 error by 0.0072 compared to the strongest baseline (FontDiffuser).
• Qualitative Performance: Visual comparisons demonstrated that InkDiffuser generates characters with superior structural consistency and clearer stroke rendering compared to GAN-based (e.g., CF-Font, IF-Font) and other diffusion-based methods (e.g., FontDiffuser, MXFont++). The generated outputs exhibited more authentic ink diffusion and fewer artifacts like missing strokes or blurriness.
• Ablation Studies:
  • The STAF module was shown to significantly improve structural metrics (SSIM, L1) by restoring missing topological details.
  • The DIS loss was critical for improving perceptual quality (LPIPS) and visual authenticity, specifically in synthesizing realistic ink seepage.
  • The combination of both modules yielded the best results, with STAF providing structural sharpness and DIS softening these edges to match natural ink flow.
• Hyperparameter Sensitivity: Experiments indicated that a DIS loss weight ($\lambda_{dis}$) of 0.02 provided the optimal trade-off between structural fidelity and ink realism. Higher weights caused the model to prioritize style over content, leading to structural degradation.

Significance and Claims

The paper claims that InkDiffuser represents a significant advancement in Chinese calligraphy synthesis by directly addressing the gap between digital generation and the physical reality of brush writing. By explicitly modeling high-frequency structural cues and differentiable ink morphology, the method overcomes the "flat" or "digital" appearance of previous works. The authors assert that their approach achieves state-of-the-art performance in both structural accuracy and visual ink realism, particularly for complex characters where stroke interactions are dense. The work establishes that incorporating explicit morphological priors into diffusion models is a viable and effective strategy for generating high-fidelity artistic styles that require fine-grained control over texture and boundary dynamics.