Can We Change the Stroke Size for Easier Diffusion?

1. Problem Statement

Diffusion models, particularly in the Denoising Diffusion Probabilistic Model (DDPM) framework, face significant challenges during the low signal-to-noise ratio (SNR) regime (high timesteps).

• The Core Issue: At high timesteps, the input image is dominated by noise, yet the model is trained to predict pixel-level noise (high-frequency details). This creates a difficult regression problem where the model must infer fine-grained structures from an input containing almost no signal.
• Consequences:
  • Optimization Difficulty: The high-frequency targets lead to large stochastic gradient variance, slowing down convergence.
  • Reverse-Chain Fragility: Errors made during early sampling steps (high noise) accumulate as high-frequency artifacts in the final generated image.
• Analogy: The authors compare standard diffusion to an oil painter using only the finest brush throughout the entire process, which is inefficient for laying down global structure. They propose mimicking the human artist's workflow: using coarse strokes first for global structure, then fine strokes for details.

2. Methodology: MultiStroke

The paper introduces MultiStroke, a controlled intervention that dynamically adjusts the "effective stroke size" (spatial granularity) of the supervision targets, predictions, and perturbations across timesteps.

A. The Stroke Operator ($S_k$)

The core mechanism is a linear operator $S_k$ defined as average pooling (kernel size $k \times k$, stride $k$) followed by nearest-neighbor upsampling back to the original resolution.

• Mathematical Property: $S_k$ acts as an orthogonal projector onto a block-constant subspace. It induces an exact decomposition of the image space into:
  • Coarse Subspace ($H_c$): Block averages (low-frequency/global structure).
  • Detail Subspace ($H_d$): Within-block variations (high-frequency details).

B. The Control Schedule ($w_t$)

A timestep-dependent weight $w_t \in [0, w_{max}]$ controls the degree of coarsening.

• Early Timesteps (High Noise): $w_t$ is high. The model operates in a "coarse" space, suppressing high-frequency details.
• Late Timesteps (Low Noise): $w_t$ decays to 0. The model reverts to standard pixel-level prediction to refine details.

C. Training Objective (Stroke-Space Loss)

Instead of predicting raw noise $\epsilon$, the model predicts a transformed target.

• Input Transformation: The noisy input $x_t$ is mixed: $x^{ms}_t = (1-w_t)x_t + w_t S_k(x_t)$.
• Target Transformation: The noise target $\epsilon$ is similarly mixed: $\tilde{\epsilon}(t) = (1-w_t)\epsilon + w_t S_k(\epsilon)$.
• Loss Function: The model minimizes the squared error between the transformed prediction and the transformed target:
  $$L_{MS} = \mathbb{E}[\| (1-w_t)(\epsilon_\theta - \epsilon) + w_t S_k(\epsilon_\theta - \epsilon) \|^2]$$
• Key Insight: This does not change the optimal population predictor (the conditional mean remains the same) but reweights the optimization path, reducing the complexity of the regression target during high-noise steps.

D. Sampling Process

During inference, the reverse chain is modified to maintain consistency with the training objective:

1. State Smoothing: The current state $x_t$ is smoothed via the stroke operator before being fed to the network.
2. Noise Injection: The injected Gaussian noise $\eta_t$ is also smoothed (mixed) before being added to the update step.
   This ensures the reverse chain evolves in the same "roughness" space as the training objective, preventing the accumulation of high-frequency errors early in the generation process.

3. Theoretical Analysis

The authors provide a rigorous operator-theoretic framework:

• Target Complexity Reduction: They prove that MultiStroke reduces the effective complexity of the regression target. Specifically, the variance of the "detail" component of the noise target is reduced by a factor of $(1-w_t)^2$.
• Detail-Energy Contraction: Under an affine surrogate model of the reverse dynamics, they derive a recursion showing that the energy of propagated detail errors and injected detail noise is explicitly contracted by $(1-w_t)$ and $(1-w_{t-1})$ respectively.
• Bias-Stability Trade-off: While the method stabilizes the reverse chain, aggressive early coarsening can introduce a "background bias" (under-specified details) that later fine-tuning steps may only partially recover.

4. Key Contributions

1. Mechanism Identification: Formalized stroke-size control as a single operator applied consistently to training inputs, targets, and sampling perturbations.
2. Operator-Theoretic Framework: Proved that the pooling-plus-upsampling operator is an orthogonal projector, enabling exact coarse/detail decomposition and deriving the Fourier attenuation envelope.
3. Mechanism Validation: Demonstrated that the intervention changes the optimization path and reverse-chain dynamics without altering the population solution, effectively acting as a frequency-aware regularizer.

5. Experimental Results

Experiments were conducted on CIFAR-10, CelebA-HQ, and FFHQ with compute-matched baselines.

• Optimization Stability: MultiStroke achieves lower training loss with comparable gradient norms, indicating easier optimization.
• Quality Improvements (Compute-Limited):
  • CIFAR-10: Lower FID (14.43 vs 20.16 at 40k steps) and higher One-Class Score (OCS).
  • CelebA-HQ/FFHQ: Significant FID improvements under tight sampling budgets (10 and 20 steps). For example, on CelebA-HQ with 10 steps, FID dropped from 183.71 (DDPM) to 110.44 (MultiStroke).
• Artifact Suppression:
  • High-Frequency SNR: MultiStroke shows significantly higher Signal-to-Noise Ratio in the high-frequency band, indicating fewer spurious high-frequency artifacts.
  • Visual Quality: Generated images exhibit better global structure and fewer "grainy" artifacts, especially in low-step sampling scenarios.
• Ablation Studies: Confirmed that both training and sampling must use the stroke control for optimal FID; using it only at inference or only at training yields suboptimal results.

6. Significance and Conclusion

• Novelty: Unlike previous works that focus on loss reweighting (e.g., Min-SNR) or architectural changes, MultiStroke explicitly manipulates the spatial scale of the learning target.
• Efficiency: It offers a lightweight, plug-and-play mechanism that improves sample quality, particularly in low-step sampling regimes where diffusion models typically struggle with artifacts.
• Implication: The work suggests that treating diffusion as a multi-scale process (coarse-to-fine) rather than a uniform pixel-level regression problem is a more effective strategy for handling high-noise regimes. It provides a principled way to trade off bias (potential loss of fine detail) for stability (reduction of high-frequency artifacts).

In summary, MultiStroke successfully adapts the "coarse-to-fine" intuition of human art to diffusion models, mathematically proving that attenuating high-frequency details during high-noise steps leads to more stable training and higher-quality generation with fewer artifacts.