🤖 machine learning

StippleDiffusion: Capacity-Constrained Stippling using Controlled Diffusion

StippleDiffusion introduces the first diffusion-based sampler that combines a learned point-distribution prior with image-defined capacity constraints via a specialized ControlNet architecture, enabling fast, differentiable, and generalizable stippling that matches the quality of traditional iterative optimizers while supporting arbitrary target densities and point budgets.

Original authors: Ofir Gilad, Aleksander Plocharski, Przemyslaw Musialski, Andrei Sharf

Published 2026-05-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ofir Gilad, Aleksander Plocharski, Przemyslaw Musialski, Andrei Sharf

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a black-and-white photo and you want to recreate it using only dots. This is called "stippling." The rule is simple: where the photo is dark, you need a lot of dots packed tightly together. Where the photo is light, you need very few dots, spaced far apart.

For decades, creating these dot patterns was like trying to solve a giant, complex math puzzle every single time you wanted to make a new picture. You had to start from scratch, move dots around slowly, and check if they were too close or too far apart. It was slow, and if you wanted to change the picture or the number of dots, you had to run the whole puzzle again.

StippleDiffusion is a new tool that solves this problem using a "smart guesser" (a type of AI called a diffusion model). Here is how it works, using some everyday analogies:

1. The "Blueprint" vs. The "Construction Site"

Traditional methods are like a construction crew that has to measure every single brick and lay it down one by one for every new building. If you want a different building, they have to start over.

StippleDiffusion is like a master architect who has already learned the rules of how bricks should look. Instead of building from nothing, it starts with a rough pile of bricks (dots) that are already in the right general neighborhood. It then acts like a skilled editor, gently nudging the bricks into their perfect spots.

2. The "Traffic Cop" and the "Crowd"

The biggest challenge in stippling is keeping the dots from clumping together (like a crowd of people tripping over each other) while still making sure there are enough of them in the dark areas of the image.

The paper introduces a special "Traffic Cop" (a technical component called a ControlNet with a "Gated" mechanism).

The Problem: If you just tell a computer "put dots here," it might ignore the rule that dots shouldn't touch, causing them to clump up like a messy pile of marbles.
The Solution: The Traffic Cop doesn't just shout orders; it gently guides the dots. It says, "Okay, move toward the dark area, but remember your personal space bubble." This ensures the dots stay evenly spaced (a property called "blue noise") while still following the shape of the image.

3. The "SDEdit" Shortcut (The "Edit, Don't Start Over" Trick)

Usually, AI image generators start with a screen full of static noise (like a TV tuned to a dead channel) and slowly clean it up to reveal an image. This takes a long time.

StippleDiffusion uses a shortcut. Instead of starting with static noise, it starts with a "rough draft" that is already mostly correct (a density-weighted rejection sample). Think of it like editing a draft essay rather than writing one from scratch.

It skips the first 70% of the "cleaning" process.
It only focuses on the final 30% of refining the details.
This makes it much faster and prevents the dots from getting confused at the beginning.

4. The "Magic Budget"

One of the coolest features is that this tool is flexible with its "budget."

Old Way: If you trained a robot to place 1,000 dots, and then asked it to place 2,000 dots, it would get confused or you'd have to retrain it.
StippleDiffusion: You can train it once with 1,024 dots. Then, at the end, you can ask it to generate 256 dots, 1,024 dots, or even 2,304 dots, and it just works. It understands the concept of the density, not just the specific number of dots.

The Result

The paper shows that this new method creates dot patterns that look just as good as the slow, old math-based methods. In fact, on a test with 1,000 different images, it matched the quality of the best traditional methods in every category (how smooth the dots look, how well they follow the dark/light areas, and how evenly they are spaced).

In short: StippleDiffusion is a smart, fast way to turn images into dot art. It learns the rules of dot placement once, and then can instantly apply those rules to any new image, any level of darkness, and any number of dots, without needing to solve a math puzzle every time.

Technical Summary: StippleDiffusion

Problem Statement

The generation of high-quality 2D point sets with prescribed spatial properties, specifically for image stippling, is a fundamental challenge in computer graphics. Traditional approaches rely on iterative optimization (e.g., Weighted Voronoi Stippling, Blue Noise through Optimal Transport) or specialized sampling strategies to produce structured distributions like blue noise. While effective, these methods are computationally expensive, non-differentiable, and require a fresh optimization run for every new target density or point budget. This lack of generalization precludes interactive use and large-scale generation.

Recent learned alternatives using denoising diffusion probabilistic models (DDPMs) have shown promise in learning complex geometric distributions. However, applying diffusion to scattered point sets is challenging due to their inherent convolutional structure designed for grids. Furthermore, existing diffusion methods have largely addressed unconditional point generation. They struggle to satisfy capacity-constrained, image-conditioned stippling—the requirement to generate point sets that simultaneously adhere to a learned local point-distribution prior (e.g., blue noise) and a continuous, image-defined density map. Standard conditioning architectures often lead to instability, geometric collapse, or uneven density when jointly enforcing global allocation from noise and high-frequency local constraints.

Methodology

The authors propose StippleDiffusion, the first diffusion-based sampler capable of satisfying a learned local point-distribution prior and a continuous, image-defined capacity constraint simultaneously at inference without per-density optimization.

Core Architecture

The method builds upon an Optimal Transport (OT) grid point-set diffusion baseline (Doignies et al., 2023).

OT Representation: Point sets are mapped to a regular $N \times N$ grid via optimal transport. Each grid cell stores the 2D offset between the stratum center and its assigned sample. This transforms the point set into a two-channel image, allowing standard 2D convolutions to act as approximate nearest-neighbor operators.
ControlNet Branch: To enable spatial conditioning without retraining the base model, a trainable ControlNet branch is added parallel to the frozen U-Net.
Gated Zero Convolutions: Instead of standard Zero Convolutions, the authors employ a sigmoid-gated $1 \times 1$ projection. This mechanism dynamically modulates injected features, preventing the early-stage loss of geometric regularity and preserving the base model's learned blue-noise structure under hard density signals.
Feature Extraction: The conditioning input aggregates static grid features and dynamically sampled image features. Using a GECCO-inspired approach, a lightweight CNN extracts continuous features from the high-resolution input image, which are then bilinearly interpolated at the exact floating-point coordinates of the current noisy point set. These are concatenated with static OT offsets and the target density map to form a unified condition vector.

Inference Strategy

The inference process is designed as a guided refinement rather than generation from pure noise:

Structured Initialization: Instead of starting from pure Gaussian noise, the process begins with a structural prior generated via density-weighted rejection sampling. This provides a spatially informed initial point distribution.
Truncated Diffusion (SDEdit): The model operates on a truncated diffusion schedule, focusing on the late-stage denoising regime (the final 30% of timesteps). The network learns to refine a partially noised structural prior rather than generating a distribution from scratch.
Resampling Flow: To enforce strict capacity constraints and un-clump the distribution, the method employs a RePaint-style forward-backward resampling flow. During the reverse step, the sample is diffused back to a previous timestep before predicting the next, providing multiple opportunities to refine the distribution within a local window.

Key Contributions

First Diffusion-Based Capacity-Constrained Sampler: The paper presents the first framework for stippling that accepts arbitrary target densities at inference using a single trained model, contrasting with optimization-based methods that require re-running for each density.
OT-Grid Formulation for Constrained Learning: By reinterpreting structured point generation through an optimal transport formulation, the method enables the learning of blue-noise-like distributions under constraints. The inference cost is nearly independent of the output point count, and gradients are available end-to-end.
Novel Architectural and Sampling Techniques:
- Gated Zero Convolutions: A mechanism to safely inject high-frequency spatial conditions without destroying the base model's point-repulsion capabilities.
- Truncated SDEdit Schedule: Restricting training and inference to late-stage denoising, initialized from a density-weighted rejection sample, stabilizes generation under strict constraints.
Comprehensive Evaluation: The authors introduce a quantitative and qualitative evaluation protocol adapted from blue-noise and capacity-constrained sampling literature to compare their method against prior unconditional samplers and optimization-based baselines.

Results

The method was evaluated on the Icons-50 benchmark and various synthetic density tests.

Performance Parity: On the Icons-50 test set, the learned sampler reaches parity with per-density-optimized baselines (WVS, BNOT, GBN) across all reported metrics: CVT Energy, Capacity Constraint Fulfillment, Earth Mover's Distance (EMD), Sinkhorn Distance, and Spatial Measure ( $\rho$ ).
Generalization: The single trained checkpoint (trained on 1024 points) successfully generalizes to point budgets not seen during training (e.g., 256, 576, 2304 points) and handles analytic densities outside the training distribution, including highly non-linear stress tests.
Visual Quality: Qualitative comparisons show that the method produces high-quality blue noise distributions that track global target densities while maintaining local point repulsion, avoiding the clustering and uneven density often seen in standard conditioned diffusion.
Runtime: While the iterative denoising process results in a higher fixed cost compared to deterministic samplers like WVS, the runtime remains nearly constant across different point counts, unlike optimization-based methods where time scales significantly with point count.

Significance and Claims

The paper claims to bridge the gap between example-based geometric sampling and image-conditioned generation. Its primary significance lies in demonstrating that diffusion models can be adapted for capacity-constrained stippling without the computational overhead of iterative optimization for each new target.

The authors modestly note that while their approach stabilizes diffusion-based point generation under structured constraints, it is not a replacement for deterministic low-discrepancy samplers in time-critical applications due to its iterative nature. Additionally, the method's scalability to higher dimensions is limited by the $O(k^d)$ storage cost of the OT-grid representation, a limitation inherited from the underlying baseline. However, for 2D stippling, the method offers a differentiable, generalizable, and high-quality alternative to traditional optimization techniques.