NeuralRemaster: Phase-Preserving Diffusion for Structure-Aligned Generation

NeuralRemaster introduces Phase-Preserving Diffusion (ϕ-PD), a model-agnostic method that replaces standard Gaussian noise with phase-preserving, magnitude-randomized noise to enable structure-aligned image and video generation without architectural changes or inference-time costs.

The Gist

Here is an explanation of the paper "NeuralRemaster: Phase-Preserving Diffusion" using simple language and creative analogies.

The Big Idea: Painting Without Moving the Furniture

Imagine you have a photograph of your living room. You want to use AI to change the style of the room—maybe turn it into a "cyberpunk" scene or a "watercolor painting."

The Problem with Current AI:
Most current AI tools work like a chaotic storm. They take your photo, smash it into a pile of dust (noise), and then try to rebuild it from scratch based on your description.

• The Issue: Because they smash it so hard, they lose the "blueprint" of the room. When they rebuild it, the AI might decide the sofa should be on the ceiling, or the door should be in the middle of the floor. The texture changes (it looks like a painting), but the structure (where things are) gets messed up.
• The Old Fix: To stop this, engineers usually build a massive "scaffolding" around the AI (like ControlNet). They add extra parts to the machine to force it to remember where the walls are. This makes the AI slower, heavier, and harder to use.

The New Solution (Phase-Preserving Diffusion):
The authors of this paper realized they didn't need to smash the photo into dust. They needed a smarter way to "shake" the image.

They discovered a secret in how images work: Images have two parts.

1. Magnitude (The Texture): This is the color, the grain, the "feel" of the paint.
2. Phase (The Skeleton): This is the geometry, the edges, the "skeleton" that tells the brain where the sofa and the door are.

The Analogy: The DJ and the Dance Floor
Think of an image like a song playing at a club.

• The Magnitude is the volume and the beat (the energy).
• The Phase is the rhythm and the timing (the structure).

Current AI turns the volume up and down randomly and messes with the timing, so the song becomes unrecognizable noise.
This new method (ϕ-PD) acts like a DJ who keeps the timing (Phase) exactly the same but completely changes the volume and beat (Magnitude).

• The result? The song still has the exact same rhythm and structure (the room layout stays perfect), but the sound is completely new (the style changes).

How It Works (The "Magic Trick")

1. Don't Break the Skeleton: Instead of adding random noise that destroys the image's shape, the AI adds a special kind of "structured noise."
2. Keep the Blueprint: It takes the "Phase" (the blueprint) from your original photo and locks it in place.
3. Randomize the Rest: It replaces the "Magnitude" (the texture) with random noise.
4. The Result: When the AI cleans up the noise, it naturally reconstructs the image with the exact same layout as your original, but with a brand-new look.

Why Is This Better?

• No Heavy Machinery: You don't need to add extra "scaffolding" (extra parameters) to the AI. It works with the existing AI models you already have. It's like upgrading the engine of a car without adding a trailer.
• It's Fast: Because it doesn't need to calculate extra steps or run extra modules, it runs just as fast as the original AI.
• You Control the Rigidity: The authors added a "dial" (called the Frequency Cutoff).
  • Turn the dial to "Strict": The AI keeps the skeleton 100% rigid. Great for turning a sketch into a photo without moving the lines.
  • Turn the dial to "Creative": The AI is allowed to wiggle the skeleton a little bit. Great for artistic re-imagining where you want some freedom.

Real-World Examples from the Paper

1. Turning Sketches into Photos: You can draw a rough stick-figure city, and the AI will turn it into a photorealistic city, but the buildings will stay exactly where you drew them.
2. Video Games to Real Life: They used this to make video game footage (from a simulator called CARLA) look like real-world driving footage.
   • Why it matters: Self-driving cars are trained in simulators because it's safe. But simulators look fake, so the car doesn't know how to drive in the real world. This method makes the simulator look exactly like the real world (preserving the road lines and car shapes) so the AI driver learns faster. They improved the AI driver's performance by 50%.
3. Video: It works on videos too, keeping the cars and people in the same spot frame-by-frame, so the video doesn't jitter or warp.

The Bottom Line

This paper is a "lightbulb moment" for AI image generation. It realized that to change the look of an image without changing its shape, you don't need to build a bigger, more complex machine. You just need to be smarter about how you shake the image.

By keeping the "Phase" (the skeleton) and only changing the "Magnitude" (the skin), they created a tool that is faster, cheaper, and more accurate than the current state-of-the-art methods. It's like remodeling a house without knocking down the walls.

Technical Summary

Here is a detailed technical summary of the paper "NeuralRemaster: Phase-Preserving Diffusion for Structure-Aligned Generation."

1. Problem Statement

Standard diffusion models excel at unconditional or text-to-image generation but struggle with structure-aligned generation (image-to-image tasks where spatial layout, geometry, and object boundaries must remain fixed while appearance changes). Examples include neural rendering, stylization, and sim-to-real transfer for autonomous driving.

• The Core Issue: Standard diffusion corrupts data using Gaussian noise. In the frequency domain, Gaussian noise has both random magnitudes and random phases. Since signal processing theory dictates that phase encodes spatial structure while magnitude encodes texture, standard diffusion destroys the geometric coherence of the input image.
• Current Limitations: Existing solutions (e.g., ControlNet, T2I-Adapter) attempt to fix this by adding auxiliary neural network branches to inject structural constraints. While effective, these methods introduce significant computational overhead, additional parameters, and architectural complexity, making structure-aligned generation unnecessarily difficult.

2. Methodology: Phase-Preserving Diffusion ($\phi$-PD)

The authors propose a model-agnostic reformulation of the diffusion process that operates in the frequency domain, eliminating the need for architectural changes.

A. Core Concept: Phase Preservation

Instead of corrupting the input image with standard Gaussian noise, $\phi$-PD constructs structured noise that preserves the input image's phase spectrum while randomizing its magnitude spectrum.

• Mechanism: Given an input image $I$ with Fourier transform $F_I = A_I \cdot e^{j\phi_I}$, the method constructs noise $\hat{\epsilon}$ by pairing the input phase $\phi_I$ with a random magnitude $A_\epsilon$ (derived from Gaussian noise statistics).
• Result: The noise injected into the diffusion process retains the spatial structure (phase) of the original image, ensuring that the sampling trajectory naturally aligns with the input geometry.

B. Frequency-Selective Structured (FSS) Noise

To provide a trade-off between strict structural alignment and creative flexibility, the authors introduce FSS noise.

• Control Parameter: A single frequency-cutoff radius $r$.
• Operation: A frequency mask $M(u,v)$ is applied. Frequencies within radius $r$ retain the input phase (preserving structure), while frequencies outside $r$ use random phase (allowing for texture/style variation).
• Flexibility: By adjusting $r$, users can continuously control the "rigidity" of the structure. $r=0$ behaves like standard diffusion; $r=\infty$ preserves the structure perfectly.

C. Training and Inference

• Model Agnostic: The method works with any diffusion model (DDPM or Flow Matching) and any architecture (UNet, DiT).
• Training: The model is fine-tuned using the structured noise formulation. The loss function remains the standard Mean Squared Error (MSE) between the predicted and ground-truth velocity/noise, but the expectation is taken over structured noise rather than Gaussian noise.
• Inference:
  1. Structured SDEdit: Add structured noise to the source image at a specific timestep and denoise.
  2. From Noise: Generate entirely from structured noise where the phase is derived from the input, allowing for significant appearance changes while keeping geometry fixed.
• Video Extension: The method extends to video by constructing phase-preserving noise frame-by-frame, compatible with video diffusion models without architectural changes.

3. Key Contributions

1. Phase-Preserving Diffusion Process: A novel diffusion formulation that replaces Gaussian noise with phase-preserving structured noise, maintaining spatial structure without modifying the network architecture or adding parameters.
2. Frequency-Selective Structured (FSS) Noise: A mechanism enabling continuous control over structural rigidity via a single frequency-cutoff parameter.
3. Unified and Efficient Framework: The approach is compatible with both image and video generation (DDPM and Flow Matching) and incurs zero inference-time overhead and zero additional parameters.

4. Experimental Results

The method was evaluated on three tasks using SD 1.5, FLUX-dev, and Wan 2.2-14B models:

• Photorealistic Re-rendering (UnrealCV):

  • $\phi$-PD significantly outperformed baselines (ControlNet-Tile, PNP, SDEdit) in structure preservation.
  • Metrics: Achieved a ~90% improvement in LPIPS (perceptual similarity) compared to standard SDEdit while maintaining high CLIP text alignment.
  • Qualitative: Successfully maintained object boundaries and geometry where other methods (like QWen-Edit) distorted shapes or FLUX-Kontext failed to improve photorealism.

• Stylized Re-rendering:

  • $\phi$-PD produced visually coherent stylizations that preserved scene structure, whereas prior methods often introduced geometric distortions.
  • Metrics: Highest SSIM (0.7842) and lowest ABSREL (0.8087) among compared methods.

• Simulation Enhancement (Sim-to-Real for Autonomous Driving):

  • Applied to CARLA simulator videos to enhance them for Waymo Open Dataset planning.
  • Impact: Improved planner transfer performance by 50% in a zero-shot setting, significantly narrowing the sim-to-real gap.
  • Video Quality: Maintained consistent road boundaries and vehicle shapes across frames, unlike comparison methods which produced artifacts.

• Efficiency:

  • Parameters/FLOPs: 0% increase (unlike ControlNet which adds ~50% parameters).
  • Inference Time: Comparable to standard SDEdit (~20s/image), significantly faster than inversion-based methods like PNP or FBSDiff.

5. Significance and Conclusion

• Paradigm Shift: The paper challenges the assumption that structure-aligned generation requires complex architectural adapters. It demonstrates that the inefficiency lies in the diffusion process (destroying phase) rather than the network architecture.
• Practical Impact: By removing the need for extra parameters and inference overhead, $\phi$-PD makes high-fidelity, structure-aligned generation accessible and efficient for real-world applications like autonomous driving simulation, video editing, and 3D rendering.
• Future Potential: The method is orthogonal to existing conditioning techniques and can be combined with them. It opens avenues for tasks like deblurring, relighting, and super-resolution where structural integrity is paramount.

In summary, NeuralRemaster provides a mathematically grounded, lightweight, and highly effective solution for structure-aligned generation by leveraging the fundamental property that phase preserves structure.