💻 computer science

StructDiff: A Structure-Preserving and Spatially Controllable Diffusion Model for Single-Image Generation

This paper presents StructDiff, a novel single-image generation framework that leverages an adaptive receptive field and 3D positional encoding to overcome existing limitations in structural preservation and spatial controllability, while introducing an LLM-based evaluation metric to assess generation quality.

Original authors: Yinxi He, Kang Liao, Chunyu Lin, Tianyi Wei, Yao Zhao

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yinxi He, Kang Liao, Chunyu Lin, Tianyi Wei, Yao Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a single, beautiful photograph of a cat sitting on a windowsill. You want to create hundreds of new, unique photos of that same cat, but maybe sitting in different spots, looking slightly different, or even with a different background, all while keeping the cat's unique fur pattern and the window's shape exactly right.

This is the challenge of Single-Image Generation. Most AI models today are like students who memorized a whole library of books; they can draw a cat, but they might mix up details from a tiger or a lion because they've seen too many different cats. They don't know your specific cat's quirks.

The paper introduces StructDiff, a new AI tool designed to be the ultimate "photocopier" that doesn't just copy, but creatively remixes a single photo while keeping its soul intact. Here's how it works, broken down into simple concepts:

1. The Problem: The "Blurry Copy" and the "Rigid Stencil"

Previous methods had two main flaws:

The Multi-Scale Approach (The "Zoom-Out" Problem): Imagine trying to draw a city by first sketching the whole continent, then the country, then the city, then the street. If you make a mistake at the continent level, the whole city is wrong. Older AI models did this, leading to "error accumulation" where the final image looks distorted.
The Single-Scale Approach (The "Fixed Lens" Problem): Other models tried to look at the whole image at once with one fixed lens. The problem? A lens that sees the whole room is too blurry to see the texture of the cat's whiskers. A lens that sees the whiskers is too zoomed in to see the whole cat. They couldn't do both at once.

2. The Solution: The "Adaptive Eye" (ARF)

StructDiff solves this with a module called the Adaptive Receptive Field (ARF).

The Analogy: Think of a human artist looking at a painting. When they paint the background, they step back to see the whole room (a wide view). When they paint the cat's eye, they lean in close (a narrow view).
How StructDiff does it: Instead of having different cameras for different zoom levels, StructDiff has one camera that can instantly change its "lens" depending on what it's looking at. If it sees a big, rigid object (like a building or a whole animal), it uses a "wide lens" to keep the shape straight. If it sees fine details (like fur or leaves), it switches to a "macro lens" to capture the texture.
The Result: It creates images that are sharp everywhere, preserving the big structures without losing the tiny details.

3. The Superpower: The "GPS Tag" (3D Positional Encoding)

This is the paper's biggest innovation. Usually, if you ask an AI to move a cat from the left side of the photo to the right, the AI gets confused and might stretch the cat or leave a weird ghost behind.

The Analogy: Imagine every pixel in your photo has a tiny, invisible GPS tag attached to it. This tag tells the AI two things:
1. Where am I? (The X and Y coordinates).
2. Who am I? (Am I part of the cat [foreground] or the wall [background]?).
How it works: Because the AI knows exactly where every "piece" of the image belongs, you can tell it: "Move the cat 5 inches to the right." The AI simply updates the GPS tags for the cat pixels and leaves the wall pixels alone.
The Magic: You can resize the cat, move it, or even change just its nose shape, and the rest of the image stays perfectly stable. It's like having a digital puppet master that can move parts of the photo without tearing the fabric.

4. The New Judge: The "AI Critic" (LLM Evaluation)

How do you know if the new photos are good?

The Old Way: Use math formulas (metrics) that count pixel differences. These are often bad at understanding "beauty" or "weirdness." Or, hire 50 humans to look at pictures and vote, which takes forever and costs a lot of money.
The New Way: The authors taught a Large Language Model (like a super-smart AI chatbot) to be an art critic. They showed the AI the original photo and the new generated photos, asking it to rate them on "How much does it look like the original?" and "Is it blurry or weird?"
The Result: This AI critic agreed with human judges 92.5% of the time, but it did it instantly and for free. It's a fast, cheap, and surprisingly accurate way to test new AI art tools.

5. What Can You Do With It?

Because StructDiff is so flexible, it can do many cool things without needing to be retrained:

Text-to-Image: "Make the cat look like a tiger."
Outpainting: Extend the photo to show what's behind the cat.
Editing: Change the cat's expression or move the window.
Style Transfer: Keep the cat's shape but make it look like a watercolor painting.

Summary

StructDiff is like a master forger who doesn't just copy a painting; they understand the brushstrokes, the lighting, and the composition so well that they can recreate the painting in a million different ways, move the subjects around, and even change the style, all while keeping the original masterpiece's "soul" perfectly intact. It solves the problem of keeping the big picture clear while seeing the small details, and it gives us a new, smart way to judge if the art is actually good.

1. Problem Statement

Single-image generation aims to synthesize diverse samples that share the visual content and internal statistics of a single input image without relying on external datasets. While existing methods (e.g., SinGAN, SinDiffusion) have made progress, they face two critical limitations:

Structural Preservation: Existing approaches struggle to maintain the global layout, especially for images containing large rigid objects. Multi-scale methods suffer from error accumulation during hierarchical generation, while single-scale methods often fail to capture both fine textures and global structures simultaneously due to fixed receptive fields.
Lack of Spatial Controllability: Current methods lack mechanisms to explicitly guide the position, scale, or placement of generated content. Users cannot easily manipulate specific regions (e.g., shifting an object or adjusting facial features) without retraining or losing structural coherence.
Evaluation Gaps: Existing objective metrics (like SIFID, LPIPS) often fail to correlate with human perception in single-image tasks, while user studies are too labor-intensive.

2. Methodology: StructDiff

StructDiff is a generative framework based on a single-scale diffusion model (derived from DDPM) designed to address the above challenges through three core innovations:

A. Adaptive Receptive Field (ARF) Module

To solve the structural preservation issue within a single-scale framework, StructDiff introduces an Adaptive Receptive Field module.

Architecture: It removes downsampling, upsampling, and attention modules from the standard UNet to prevent "memorization" (overfitting to the single image).
Mechanism: The ARF block consists of multiple parallel convolutional branches with different kernel sizes (e.g., 5×5, 7×7, 9×9, 11×11).
Dynamic Fusion: An attention mechanism dynamically assigns weights to these branches based on the image content. For large foreground structures, branches with larger receptive fields receive higher weights to capture global context; for fine details, smaller kernels are prioritized. This allows the model to adaptively balance local texture modeling and global structure preservation without hierarchical error accumulation.

B. 3D Positional Encoding (PE) for Spatial Control

To enable spatial controllability, StructDiff incorporates a novel 3D positional encoding as an explicit spatial prior.

Vector Construction: The encoding vector at each pixel combines spatial coordinates $(x, y)$ and a binary foreground-background mask ( $m$ ).
Fourier Embedding: The vector is transformed using a learnable Fourier mapping ( $\sin(B_{fo} \cdot [x', y', m]^T)$ ). This ensures translation equivariance (preserving consistency in uncontrolled regions) and high-frequency sensitivity (maintaining sharp boundaries).
Control: During inference, users can modify the PE (e.g., shifting coordinates or changing the mask) to guide the generation process. This allows for operations like translating objects, scaling regions, or fine-grained editing (e.g., adjusting facial features) while maintaining the overall structural layout.

C. Training-Free Applications

StructDiff supports various downstream tasks without retraining by utilizing specialized guidance mechanisms:

Text-Guided Generation: Uses CLIP-based gradient guidance to align generated content with text prompts.
Image Editing/Outpainting: Uses a sampling strategy that injects low-frequency structures from a reference image or the source image itself into masked regions to ensure seamless extension and structural consistency.

D. LLM-Based Evaluation Metric

The authors propose a new evaluation paradigm using Large Language Models (LLMs) (specifically GPT-4o) to assess generation quality.

Role: The LLM acts as an "image generation quality expert."
Criteria: It evaluates images on two dimensions: Generation Quality (blur, artifacts) and Absence of Distortion (structural consistency).
Advantage: This method offers a reliable, low-cost alternative to human user studies, showing a 92.5% agreement rate with human preferences, significantly outperforming traditional no-reference metrics like MUSIQ and DB-CNN.

3. Key Contributions

StructDiff Framework: A single-scale diffusion model that achieves multi-scale structural awareness via an Adaptive Receptive Field (ARF) module, significantly improving the preservation of large objects and global layouts.
Spatial Controllability: The first exploration of 3D Positional Encoding (with Fourier embedding) in single-image generation, enabling precise, explicit control over object location, scale, and local details.
Novel Evaluation Metric: An LLM-based scoring system that bridges the gap between inaccurate objective metrics and expensive user studies, providing a robust benchmark for structural consistency and visual quality.

4. Experimental Results

The method was evaluated on three datasets: Places50, Mulmini-N (general images), and Mulmini-L (images with large objects).

Structural Preservation: StructDiff outperformed state-of-the-art methods (SinGAN, SinDDM, SinDiffusion) in SIFID (measuring internal statistics capture) and visual quality metrics (MUSIQ, DB-CNN). It showed particular superiority on Mulmini-L, where it maintained the integrity of large rigid objects better than competitors.
Spatial Control: In controllable generation tasks (shifting/scaling objects), StructDiff achieved significantly higher PSNR and SSIM scores for both foreground and background compared to SinDDM. It successfully avoided the blurring and artifacts seen in baseline methods.
Evaluation Validity: The LLM-based metric achieved a 92.5% agreement rate with human user studies, validating its effectiveness as a proxy for human judgment.
Ablation Studies: Confirmed that removing the ARF module or the foreground-aware loss led to fragmented structures, and that 3D PE with Fourier embedding was essential for decoupling foreground/background manipulation.

5. Significance

Paradigm Shift: StructDiff challenges the reliance on multi-scale architectures for single-image generation, proving that adaptive receptive fields within a single-scale framework can achieve superior structural consistency.
Practical Utility: By enabling spatial control without retraining, StructDiff opens new possibilities for interactive image editing, outpainting, and style transfer using only a single source image.
Evaluation Standard: The introduction of an LLM-based evaluation metric addresses a critical bottleneck in the field, offering a scalable and accurate way to assess single-image generation quality, which is notoriously difficult for traditional metrics.
Limitations: The authors acknowledge that the model struggles with complex background separations and cannot "hallucinate" occluded parts of objects that are not present in the source distribution, suggesting future work in curriculum learning.

In summary, StructDiff represents a significant advancement in single-image generation by solving the trade-off between structural fidelity and spatial controllability, while also providing a new standard for evaluating these generative capabilities.