🤖 AI

General and Efficient Steering of Diffusion Models

This paper introduces Noise-Aligned RFM Steering (NA-RFM), an efficient method for steering diffusion models that achieves faster inference and improved accuracy by combining offline-computed noise alignment and Recursive Feature Machine activation steering, thereby eliminating the need for computationally expensive per-step gradient computations.

Original authors: Qingsong Wang, Mikhail Belkin, Yusu Wang

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qingsong Wang, Mikhail Belkin, Yusu Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The Stubborn Artist

Imagine you have hired a brilliant, world-class painter (this is the Diffusion Model). This painter is amazing at creating realistic images, but they are "unconditional." This means they don’t take orders like "Paint me a cat" or "Paint me a sunset." They just paint whatever comes to mind, usually starting with a chaotic mess of noise that slowly turns into a picture.

If you want this painter to create something specific—like a blue bird or a blonde woman—you have two traditional options, both of which are annoying:

Retrain the Painter: You spend months teaching them new rules. This is expensive and slow.
The "Nagging" Method (Gradient Guidance): As the painter works, you stand over their shoulder with a magnifying glass. Every time they put down a brushstroke, you check if it looks like a bird. If it doesn’t, you gently push their hand in the right direction. This works, but it’s incredibly slow because you have to check every single stroke.

The Solution: NA-RFM (Noise-Aligned RFM Steering)

The authors propose a new method called NA-RFM. Think of this as giving the painter a cheat sheet and a style guide before they start painting. This allows the painter to work at full speed without you nagging them during the process.

The method has two main parts, applied at different stages of the painting process:

1. The "Big Picture" Sketch (Noise Alignment)

When: At the very beginning, when the canvas is still mostly messy noise.
The Analogy: Imagine the painter is staring at a foggy window. You can’t see details yet, but you can see the general shape.
How it works: The researchers looked at thousands of pictures of the target object (e.g., birds) and calculated the "average shape" and "average color" using a statistical method called PCA. They create a simple correction signal.
The Result: When the painter starts with the foggy noise, this signal gently nudges the overall composition toward the right general vibe (e.g., "Make this look more like a bird silhouette than a car silhouette"). It’s a coarse, high-level fix that happens instantly.

2. The "Style Guide" Injection (RFM Steering)

When: In the middle and end of the process, when the image is starting to look like a real picture.
The Analogy: Now the painter is adding details. Instead of nagging them, you hand them a Style Guide that says, "Remember, birds have feathers, not fur."
How it works:

Offline Learning: Before generating any images, the researchers take labeled photos (e.g., "This is a bird," "This is not a bird") and pass them through the AI’s internal "brain" (the neural network activations).
Finding the Direction: They use a technique called Recursive Feature Machines (RFM) to find a specific "direction" in the AI’s internal data space that separates "birds" from "everything else." Think of this as finding a specific lever inside the AI’s brain that, when pulled, makes the AI think more like "bird."
Online Application: When the AI is generating a new image, the researchers simply pull that lever (add a small mathematical vector) at the right moment. They don’t need to calculate gradients or check the image step-by-step. They just inject this "bird-ness" signal into the AI’s internal thoughts.

Why Is This Better?

Speed: Because you aren’t "nagging" the AI at every step (calculating gradients), the AI can paint much faster. The paper claims it is up to 16 times faster than the previous best methods.
Quality: The images look better (lower FID score, which means they look more like real photos) and are more accurate (higher chance of actually being the object you wanted).
No Retraining: You don’t need to teach the AI new skills. You just give it this pre-computed "cheat sheet" based on a few example photos.

Real-World Examples from the Paper

The authors tested this on several tasks:

CIFAR-10 (Simple Objects): They asked the AI to generate specific classes like "cats" or "trucks." NA-RFM got 96.6% accuracy, compared to 77.1% for the previous best method.
ImageNet (Complex Objects): They tried generating specific dog breeds (like the Kuvasz) or tandem bicycles. The method successfully steered the AI toward these specific, hard-to-describe objects.
CelebA (Human Attributes): They tried to generate people with specific combinations, like "Old Male" or "Blonde Female." The method worked even for rare combinations (like "Blonde Male," which is rare in the training data).
Bird Species (Fine-Grained): They tried to generate specific bird species that weren’t even in the AI’s original training data (like the Scarlet Macaw). The AI managed to create images that looked like those specific birds, proving the method can learn from just a few example photos.

Summary

NA-RFM is like giving a talented but stubborn artist a pre-drawn sketch (for the general shape) and a specific style manual (for the details) before they start. This allows the artist to work quickly and accurately without needing constant supervision or retraining. It makes generating specific, controlled images from AI models faster and more efficient.

Technical Summary: General and Efficient Steering of Diffusion Models

Problem Statement
Steering pretrained diffusion models toward conditions that were unseen or unavailable during training—such as new object classes, fine-grained species, or specific visual properties—typically requires substantial computational overhead. Existing approaches generally fall into two categories: retraining the model with conditional inputs or employing per-step gradient computations during inference (e.g., classifier guidance or training-free gradient methods). Both strategies incur high costs; gradient-based methods require backpropagation through the diffusion model or an auxiliary classifier at every denoising step, significantly slowing down generation. The authors identify a need for a post-hoc steering method that is both general and efficient, avoiding inference-time gradients while maintaining high target accuracy and image quality.

Methodology: Noise-Aligned RFM Steering (NA-RFM)
The authors propose Noise-Aligned RFM Steering (NA-RFM), a framework that combines two offline-computed signals to steer unconditional diffusion models during sampling without gradient guidance. The method relies on two empirical observations regarding the behavior of diffusion model activations at different noise levels:

High-Noise Coarse Structure: At high noise levels, reverse trajectories contain coarse class information, but forward-process activations are not discriminative. The authors utilize a Gaussian/PCA approximation of the diffusion model in this regime.
Transferable Activation Directions: At moderate to low noise levels, forward-process activations are strongly class-discriminative, and the discriminative direction remains stable (aligned) across intermediate and late timesteps.

Based on these observations, NA-RFM operates in two phases:

Offline Preparation:
- Noise Alignment (High Noise): The method computes Principal Component Analysis (PCA) statistics for the target class and the full dataset. This allows for the construction of a pixel-space guidance signal, $g^{PCA}_t$ , defined as the difference between a class-conditional PCA denoiser and an unconditional PCA denoiser. This provides coarse control at high noise levels where forward activations are uninformative.
- RFM Direction Discovery (Intermediate/Late Noise): The authors use Recursive Feature Machines (RFMs) to learn a target-discriminative steering direction, $v_c$ , from labeled forward-process activations at a reference low-noise level. RFMs are chosen for their ability to learn task-adapted feature metrics from limited examples in high-dimensional spaces. The direction is derived from the Average Gradient Outer Product (AGOP) of the RFM predictor, ensuring it captures the most separable activation directions.
Online Inference (Gradient-Free):
- During sampling, the method applies Noise Alignment at high noise levels ( $\sigma_t \geq \sigma_{end}$ ) by adding the precomputed PCA correction to the denoised estimate.
- At intermediate and late timesteps ( $\sigma_t \in [\sigma^{lo}_R, \sigma^{hi}_R]$ ), the method applies RFM Activation Steering. This involves a lightweight activation edit where the learned direction $v_c$ is added to the activation tensor of a selected U-Net block. To amplify the effect, the method extrapolates from the base denoised estimate toward the steered estimate.
- Crucially, no step requires backpropagation through a classifier or the diffusion model. The online cost consists only of forward passes and matrix-vector products.

Key Contributions

Gradient-Free Post-Hoc Steering: NA-RFM enables controllable generation for unseen conditions without retraining the diffusion model or computing gradients during inference.
Hybrid Guidance Mechanism: The method effectively combines high-noise pixel-space PCA alignment with intermediate/late activation-space RFM steering, addressing the limitations of using a single mechanism across all noise levels.
Efficiency: By avoiding per-step gradient computations, NA-RFM achieves substantial inference speedups compared to state-of-the-art training-free gradient baselines.
Generalizability: The approach is demonstrated to work across various architectures (U-Nets, Transformers) and tasks (class guidance, multi-attribute guidance, fine-grained out-of-distribution steering).

Results
The authors evaluate NA-RFM on CIFAR-10, ImageNet, CelebA-HQ, and fine-grained bird species (Birds-525).

CIFAR-10: NA-RFM achieves 96.6% guidance accuracy with an FID of 41.4, outperforming the state-of-the-art training-free gradient baseline TFG (77.1% accuracy, FID 73.9) and noise-conditioned classifier guidance (86.0% accuracy, FID 41.9). It is 16× faster than TFG.
ImageNet: On 256×256 unconditional ADM models, NA-RFM achieves 75.8% average accuracy compared to 59.8% for TFG-4, with improved FID (98 vs. 165).
CelebA-HQ: In multi-attribute guidance (e.g., Gender+Hair, Gender+Age), NA-RFM outperforms TFG on 7 out of 8 attribute combinations, achieving 96.0% average accuracy for Gender+Age and 83.3% for Gender+Hair. It shows particular strength in rare combinations (e.g., Male+Blond).
Fine-Grained Birds: On out-of-distribution bird species, NA-RFM achieves 14.1% average accuracy compared to 2.2% for TFG-4, demonstrating the ability to steer toward specific visual categories not present in the training distribution.
Extensions: The method extends to transformer-based latent diffusion (SiT-XL/2), raising ImageNet accuracy from 12.9% to 61.3%, and to Stable Diffusion 1.5 for depth-of-field control, improving foreground/background sharpness ratios.

Significance
The paper claims that NA-RFM provides a general recipe for efficiently steering unconditional diffusion models. By decoupling the construction of guidance signals from the sampling process, it eliminates the computational bottleneck of gradient-based methods. The authors argue that this approach offers stronger target control and better image quality than existing post-hoc baselines while maintaining the speed advantages of unconditional generation. The method highlights the utility of offline-computed signals—PCA statistics for high-noise structure and RFM directions for activation-space discrimination—to achieve precise, gradient-free control over generative models.