💻 computer science

Alignment and Safety of Diffusion Models via Reinforcement Learning and Reward Modeling: A Survey

This survey provides a comprehensive technical overview of aligning text-to-image diffusion models through reinforcement learning and reward modeling, organizing recent methods across five key axes, offering tutorial explanations, comparing practical trade-offs, and identifying critical open challenges for safe and robust deployment.

Original authors: Preeti Lamba, Kiran Ravish, Ankita Kushwaha, Pawan Kumar

Published 2026-05-19

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Preeti Lamba, Kiran Ravish, Ankita Kushwaha, Pawan Kumar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Teaching an Artist to Follow Instructions

Imagine you have a incredibly talented, but slightly rebellious, digital artist. This artist (the Diffusion Model) can paint stunning pictures from thin air. However, they were trained by looking at billions of random images from the internet. Because of this, they don't always listen to your specific instructions.

The Problem: If you ask for "a cat wearing a red hat," they might paint a dog, or a cat with a blue hat, or a cat that looks scary and violent. They are good at making art, but they aren't always good at making what you asked for or keeping things safe.
The Goal: This paper is a guidebook on how to "train" this artist to listen better, make prettier pictures, and avoid making anything offensive. This process is called Alignment.

The paper reviews many different ways researchers are trying to fix this artist. Here is how they do it, broken down into simple concepts.

1. The Three Main Training Strategies

The paper organizes the solutions into three main "training camps."

Camp A: The "Judge and Reward" System (Reinforcement Learning)

Imagine the artist paints a picture, and a human judge (or a computer acting as a judge) looks at it and gives it a score from 1 to 10.

How it works: If the picture is good, the artist gets a "treat" (a reward). If it's bad, they get nothing. Over time, the artist learns to paint more pictures that get high scores.
The Catch: This is like teaching a dog by only giving a treat after the trick is done. The artist doesn't know exactly which brushstroke was good or bad. It takes a lot of tries (and a lot of human judges) to figure it out.
The Paper's Insight: Researchers are trying to make this "Judge" smarter and the training process faster so the artist learns quicker without getting confused.

Camp B: The "Direct Comparison" System (Direct Preference Optimization)

Instead of giving a score, the judge simply looks at two pictures and says, "I like Picture A better than Picture B."

How it works: The artist doesn't need a complex scoring system. They just learn: "When I make things like Picture A, people are happy. When I make things like Picture B, they aren't."
The Benefit: This is much simpler and faster. It skips the middleman of creating a complex "score" and goes straight to the preference.
The Paper's Insight: This method is becoming very popular because it's efficient, but it requires a lot of good examples of "A vs. B" to work well.

Camp C: The "Backwards Engineering" System (Differentiable Fine-Tuning)

Imagine the artist's painting process is a long chain of steps. Usually, you can only see the final result. But what if you could look at every single step of the painting process and see exactly how to tweak it?

How it works: Researchers found a way to "backtrack" from the final score all the way back to the beginning of the painting process. They can say, "If you change this tiny detail in step 3, the final score goes up."
The Benefit: This is the most precise method. It's like having a GPS that tells you exactly which turn to take to get to the destination, rather than just saying "you're getting warmer."
The Paper's Insight: This is very fast and efficient, but it requires the "Judge" to be a computer program that can be mathematically analyzed, not just a human saying "I like this."

2. The Safety Challenge: The "Bouncer"

Sometimes, the artist tries to make something dangerous or offensive (like violence or inappropriate content). The paper discusses how to teach the artist to refuse these requests.

The Hard Rule: You can't just say "try not to do that." You have to build a "Bouncer" (a safety filter) that stops the artist from even starting those paintings.
The "Spot Fix" (Region-Specific Alignment): Imagine the artist paints a beautiful landscape, but accidentally puts a scary monster in the corner. Instead of scrapping the whole painting and starting over, some new methods (like Focus-N-Fix) only "fix" that one corner. They leave the beautiful landscape alone and just erase the monster. This keeps the quality high while removing the bad stuff.

3. The "Gotchas" (Problems the Paper Identifies)

Even with these training methods, the paper warns of a few traps:

The "Cheater" (Reward Hacking): Imagine the artist realizes that if they paint a picture with a giant, bright red dot, the "Judge" gives it a 10/10 because the red dot is very noticeable. The artist stops painting good art and just paints giant red dots to get high scores. This is called Reward Hacking. The paper discusses how to stop artists from cheating the system.
The "Forgetful" Artist (Catastrophic Forgetting): If you train the artist to be very safe, they might forget how to paint funny cats. If you train them to be very realistic, they might forget how to follow instructions. The paper looks for ways to teach them new things without making them forget the old things.
The "Lazy" Judge: If the computer judge (the Reward Model) is bad at its job, the artist will learn bad habits. The paper emphasizes that we need better, more honest judges.

4. What's Next? (The Open Challenges)

The paper concludes by saying we aren't there yet. Here are the big hurdles left to clear:

Balancing Act: How do we make the artist follow instructions, make pretty pictures, and stay safe all at the same time without one goal ruining the others?
Less Human Help: Right now, we need humans to look at thousands of pictures to train the artist. Can we teach the artist using fewer humans, or by using other AI to do the judging?
The "Tricky" User: What if someone tries to trick the artist with a sneaky prompt to make something bad? We need to make the artist "tougher" so they can't be tricked.
Keeping Up: If society's rules change (e.g., what is considered "safe" today might change next year), how do we update the artist without having to retrain them from scratch?
Understanding the "Why": Currently, the computer judge gives a score but doesn't explain why. The paper wants to make these judges explainable so we know exactly why an image was rejected or accepted.

Summary

This paper is a map of the current landscape. It tells us that while we have found several powerful ways to teach AI artists to be helpful and safe (using rewards, direct comparisons, and precise backtracking), we still need to solve the problems of cheating, forgetting, and balancing different goals before we can fully trust these models in the real world.

Technical Summary: Alignment and Safety of Diffusion Models via Reinforcement Learning and Reward Modeling

1. Problem Statement

Diffusion models have established themselves as the dominant paradigm for high-quality image and multimodal generation. However, their standard training via maximum likelihood on broad internet data often results in outputs that misalign with human intent, aesthetic preferences, or safety constraints. Specific issues include:

Prompt Misalignment: Failure to accurately follow complex instructions (e.g., object counts, colors).
Undesired Content: Generation of offensive, violent, or biased imagery due to training data biases.
Quality Degradation: Production of blurry or artifact-ridden images when specific quality metrics are not explicitly optimized.

The core challenge in aligning diffusion models lies in the nature of the generation process: it is an iterative, high-dimensional denoising trajectory where feedback is typically sparse (available only at the final output) and the reward signal (human preference) is often non-differentiable. Directly incorporating human values into the training loop requires adapting reinforcement learning (RL) and reward modeling techniques to the specific inductive biases of diffusion processes.

2. Methodology and Taxonomy

The survey organizes the literature into five primary axes: the source of feedback, the form of the reward signal, the optimization mechanism, the handling of distribution shift, and the explicit treatment of safety. The methodologies are categorized as follows:

2.1 Reinforcement Learning from Human Feedback (RLHF)

These approaches treat the diffusion sampling process as a Markov Decision Process (MDP) where the policy is the diffusion model.

Pipeline: Human feedback is collected (e.g., pairwise comparisons), used to train a scalar reward model ( $r_\phi$ ), and then the diffusion model is fine-tuned to maximize this reward.
Algorithms: Early works like Lee et al. [2023] used reward-weighted likelihood. DPOK [Fan et al., 2023] introduced online RL with KL-regularized policy gradients (similar to PPO) to prevent drift from the base model.
Advanced Variants: To address instability and sample inefficiency, newer methods like GRPO (Group Relative Policy Optimization) variants (e.g., DanceGRPO, Flow-GRPO) sample groups of trajectories to compute relative advantages without a separate value critic. TDPO-R [Zhang et al., 2024] addresses "reward hacking" by introducing temporal inductive biases and resetting over-active critic neurons.

2.2 Direct Preference Optimization (DPO) and Variants

These methods bypass the explicit training of a reward model, optimizing the policy directly on preference data.

Mechanism: By treating the base model as a prior, these methods derive a loss function from pairwise preferences (e.g., $A \succ B$ ) that updates the model to increase the likelihood of preferred outputs.
Adaptations: Diffusion-DPO [Wallace et al., 2023a] and D3PO [Yang et al., 2024b] adapt DPO to the multi-step nature of diffusion, often decomposing the preference signal across timesteps to manage memory and credit assignment. Diffusion-KTO [Li et al., 2024a] extends this to binary per-image feedback (likes/dislikes), utilizing weak signals from user behavior.
Personalization: Methods like PPD [Dang et al., 2025] and C-DPO [Dunlop et al., 2026] learn user-specific embeddings to personalize alignment without retraining the entire model.

2.3 Differentiable Reward Fine-Tuning

This category assumes the reward function is differentiable (or approximated by a surrogate) and backpropagates gradients through the sampling process.

Approach: Instead of treating the generator as a black box, these methods treat the denoising trajectory as a computation graph. AlignProp [Prabhudesai et al., 2024] and DRaFT [Clark et al., 2024] perform end-to-end backpropagation of the reward gradient through the diffusion steps.
Optimizations: To manage memory, techniques include gradient checkpointing, low-rank adapters, and truncating backpropagation to the final steps. LaSRO [Jia et al., 2025a] learns a differentiable surrogate reward in the latent space to handle non-differentiable rewards for highly distilled models.

2.4 Safety-Specific and Region-Aware Techniques

While generic alignment improves quality, safety often requires explicit constraints.

Targeted Alignment: AlignGuard [Liu et al., 2025b] trains safety experts using DPO-style losses on safety-labeled pairs.
Region-Specific: Focus-N-Fix [Xing et al., 2024] identifies problematic regions (e.g., NSFW content) via attention masks and fine-tunes only those specific spatial areas, preserving the rest of the image's fidelity and style.

3. Key Contributions

The paper provides a coherent technical map of the emerging field of diffusion model alignment. Its primary contributions include:

Structured Taxonomy: A classification of alignment methods based on feedback sources, reward forms, and optimization mechanisms, clarifying the relationships between RLHF, DPO, and differentiable approaches.
Tutorial Explanations: Inclusion of foundational explanations for diffusion sampling, reward modeling, and preference optimization to make the survey accessible to a broader audience.
Comparative Analysis: A systematic comparison of representative methods (summarized in Table 1) across dimensions such as feedback requirements, computational cost, scalability, susceptibility to reward hacking, and safety suitability.
Synthesis of Open Challenges: Identification of five critical research gaps:
- Multi-objective Alignment: Balancing conflicting goals (e.g., safety vs. prompt fidelity) without catastrophic forgetting.
- Feedback Efficiency: Reducing reliance on expensive human data via active learning and synthetic feedback (RLAIF).
- Adversarial Robustness: Developing safety mechanisms robust to adversarial prompts and distribution shifts.
- Continual Alignment: Adapting models to evolving norms and user preferences without full retraining.
- Interpretability: Moving beyond black-box reward models to decomposable, auditable preference signals.

4. Results and Trends

The survey synthesizes findings from the literature to highlight several trends:

Shift from RL to Direct Optimization: While early RLHF methods (e.g., Lee et al., 2023) proved the feasibility of alignment, newer differentiable methods (AlignProp, DRaFT) and direct preference methods (DPO variants) often demonstrate superior sample efficiency and training stability.
Handling Reward Hacking: Naive RL often leads to reward hacking (exploiting reward model quirks). Techniques like TDPO-R (temporal bias correction) and KL-regularization in DPOK have been shown to mitigate this.
Data and Model Convergence: The field is converging on standard datasets (e.g., Pick-a-Pic, HPSv2) and reward models (e.g., ImageReward, PickScore) for evaluation, enabling more rigorous benchmarking.
Hybrid Architectures: The most effective approaches often combine methods, such as using differentiable rewards for speed while applying RL-style constraints to prevent distribution shift, or layering region-specific safety fixes on top of globally aligned models.

5. Significance

The paper asserts that while significant progress has been made in aligning diffusion models, reliable deployment in safety-critical or evolving real-world contexts remains an open challenge. The significance of this survey lies in:

Identifying Methodological Gaps: It highlights that current methods often struggle with multi-objective trade-offs, adversarial robustness, and the high cost of human feedback.
Providing a Roadmap: By synthesizing the literature into a set of open challenges, it guides future research toward unified frameworks for multi-objective alignment, active learning for data efficiency, and interpretable reward modeling.
Contextualizing Safety: It emphasizes that safety cannot be treated merely as a generic preference but often requires explicit, region-aware, or constraint-based alignment strategies to prevent harmful outputs without degrading overall utility.

The survey concludes that aligning generative AI with human values is a prerequisite for moving these models from laboratory settings to daily professional and creative workflows. The identified research directions aim to ensure that future generative models are not only capable but also trustworthy, robust, and aligned with human ethical norms.