Improving Text-to-Image Generation with Intrinsic Self-Confidence Rewards

The paper introduces SOLACE, a fully unsupervised post-training framework for text-to-image generation that leverages intrinsic self-confidence signals derived from noise recovery to optimize model performance without external reward models or annotated datasets.

The Gist

Imagine you have a brilliant but slightly chaotic artist named SD3.5. This artist is amazing at painting pictures based on your descriptions (like "a blue tree with rainbow roses"). However, sometimes the artist gets a little confused, mixes up the colors, or forgets to write the text correctly on a sign in the painting.

To fix this, the usual method is to hire a strict art critic (an external reward model) to look at every painting, give it a score, and tell the artist, "No, the tree should be greener," or "You missed the word 'Hello'." The problem is that hiring these critics is expensive, slow, and sometimes the artist learns to "game the system"—making the critic happy by painting weird, nonsensical things that just happen to get a high score, rather than actually getting better at painting.

Enter SOLACE.

The authors of this paper, Seungwook Kim and Minsu Cho, came up with a clever new idea: What if the artist could be their own critic?

The Core Idea: The "Self-Confidence" Test

Instead of hiring an outside judge, SOLACE teaches the artist to trust their own gut feeling. Here is how it works, using a simple analogy:

1. The "Denoising" Game
Imagine the artist paints a picture, but then we take a sponge and smear a little bit of random noise (static) over it.

• The Old Way: We ask an outside critic, "Is this picture good?"
• The SOLACE Way: We ask the artist, "Can you look at this smeared picture and tell me exactly what the original noise was that I added?"

2. The Confidence Score
If the artist can perfectly guess the noise they just smeared on their own work, it means they are very confident in their original painting. They know exactly what the image should look like.

• High Confidence (Good): The artist says, "I know exactly what that noise was! I'm sure my painting is right." -> Reward!
• Low Confidence (Bad): The artist stammers, "Uh, I'm not sure what that noise was... maybe I made a mistake?" -> No Reward.

Why is this a game-changer?

1. No More Expensive Critics
You don't need a team of human annotators or complex AI judges to tell the artist what to do. The artist generates the feedback themselves. It's like a musician practicing in a room and knowing instantly if they hit the right note, rather than waiting for a teacher to grade them.

2. Stopping the "Cheating"
When artists try to please a strict external critic, they often start "cheating" (reward hacking). They might paint a picture that looks weird but tricks the critic into giving a high score.
Because SOLACE uses the artist's own internal logic, the artist can't cheat. If they paint something nonsensical, they won't be able to "denoise" it themselves, so they won't get a reward. This forces them to actually improve their understanding of the world.

3. Better at the Hard Stuff
The paper shows that when the artist relies on this "self-confidence," they get surprisingly good at things that are usually hard for AI:

• Counting: Drawing exactly "four chairs" instead of three or five.
• Text: Writing "Hello" correctly on a sign instead of gibberish.
• Relationships: Putting a "cat on a mat" instead of a "cat inside a mat."

The Result: A Self-Improving Loop

Think of SOLACE as a mirror.

• The artist looks at their own work.
• They ask, "Does this make sense to me?"
• If the answer is "Yes, I can reconstruct every detail," they get a pat on the back and try to do it again.
• Over time, the artist becomes more consistent, more accurate, and better at following instructions, all without ever needing to ask for help from the outside world.

In a nutshell:
The paper introduces SOLACE, a method that lets AI image generators learn by trusting their own "gut feeling" (self-confidence) rather than relying on expensive, external judges. By asking the AI, "Can you explain your own mistakes?", the AI learns to paint better, count better, and write text better, all while avoiding the trap of trying to cheat the system.

Technical Summary

1. Problem Statement

Text-to-Image (T2I) generation has advanced significantly with diffusion and flow-matching models. However, aligning these models with human preferences, improving compositional accuracy, and ensuring text fidelity often requires post-training via Reinforcement Learning (RL). Current approaches face three major challenges:

• Reliance on External Rewards: Most methods depend on external reward models (e.g., human preference models like PickScore, OCR validators, or safety filters). These require large-scale annotated datasets, increase computational complexity (running multiple models during training), and are expensive to scale.
• Reward Hacking: Optimizing for narrow external rewards often leads to "reward hacking," where the model exploits the reward function to maximize scores while degrading other capabilities (e.g., generating nonsensical images that score high on a specific metric but fail in compositionality or realism).
• Underutilization of Intrinsic Signals: While intrinsic signals (like self-confidence) have been explored in Large Language Models (LLMs), they remain under-explored in T2I generation due to the continuous nature of denoising trajectories and the lack of discrete likelihoods.

2. Methodology: SOLACE

The authors propose SOLACE (Self-Originating LAtent Confidence Estimation), a post-training framework that replaces external critics with an intrinsic self-confidence signal derived entirely from the generative model itself.

Core Concept

The hypothesis is that a well-trained diffusion/flow-matching model possesses strong priors over real images and text-image alignment. Therefore, the model's ability to accurately recover injected noise from its own generated outputs serves as a proxy for the quality and faithfulness of that generation. High self-confidence (low reconstruction error) correlates with high-quality, aligned images.

Technical Workflow

1. Generation: Given a text prompt $c$, the policy model $\pi_\theta$ samples a group of $G$ latent outputs $\{z_0^{(i)}\}$.
2. Re-noising (Probing): Instead of decoding to pixel space, the method re-noises the generated latents $z_0^{(i)}$ at selected timesteps $t \in \mathcal{T}$ using a shared set of $K$ noise probes $\{\epsilon^{(m)}\}$.
   • $z_t^{(i,m)} = (1-t)z_0^{(i)} + t\epsilon^{(m)}$
3. Self-Denoising: The model attempts to predict the injected noise $\epsilon^{(m)}$ from the re-noised latent $z_t^{(i,m)}$.
   • The predicted noise is derived from the model's velocity field: $\hat{\epsilon}_\theta = v_\theta(z_t^{(i,m)}, t, c) + z_0^{(i)}$.
4. Reward Calculation: The reward is computed as the negative log of the Mean Squared Error (MSE) between the predicted noise and the actual injected noise.
   • $R_{SOLACE} = -\log(\text{MSE} + \delta)$
   • This scalar reward is aggregated over multiple probes and timesteps.
5. Optimization: The model is fine-tuned using Flow-GRPO (Group Relative Policy Optimization for Flow Matching). The intrinsic reward is used to calculate advantages within the group of $G$ samples, optimizing the policy to maximize self-confidence.

Key Stabilization Techniques

To prevent the model from collapsing into "reward hacking" (e.g., generating blank images that are trivial to denoise), the authors employ:

• Suffix Training: Optimization is restricted to the latter portion of the denoising trajectory (e.g., the last 60% of steps), where the denoising task remains informative but less exploitable.
• CFG-Free Scoring: While sampling uses Classifier-Free Guidance (CFG) for diversity, the self-confidence reward is calculated without CFG to ensure the base conditional model is optimized, not a guided proxy.
• Online Calculation: The reward is computed using the model currently being trained ($\pi_\theta$) rather than a fixed reference, allowing the reward signal to evolve as the model improves.

3. Key Contributions

1. Intrinsic Reward Framework: Introduction of SOLACE, the first post-training framework for T2I that relies solely on the model's intrinsic ability to recover injected noise as a reward signal, eliminating the need for external reward models or human annotations.
2. Theoretical Insight: Demonstration that a model's self-certainty (noise recovery capability) is strongly correlated with objective metrics like compositional generation, text rendering, and text-image alignment.
3. Complementarity: Proof that SOLACE can be stacked on top of models already trained with external rewards, yielding further improvements in non-target capabilities (compositionality, text) while mitigating reward hacking.
4. Efficiency: A training pipeline that avoids the computational overhead of running separate evaluators (OCR, safety, preference models) during RL.

4. Experimental Results

The method was evaluated on SD3.5-M (and SD3.5-L/FLUX.1-Dev in supplementary) across several benchmarks:

• Compositional Generation (GenEval): SOLACE significantly improved performance (e.g., from 0.65 to 0.71), nearly matching the larger SD3.5-L model despite having fewer parameters.
• Text Rendering (OCR): Achieved substantial gains in text accuracy (from 0.61 to 0.67), demonstrating that self-confidence aligns with the model's ability to render specific text strings.
• Text-Image Alignment (CLIP-Score): Improved alignment scores, indicating better adherence to prompt semantics.
• Human Preference: Showed modest but consistent improvements in human preference metrics (PickScore, HPSv2, ImageReward) without using them as training signals.
• Ablation Studies:
  • Noise Probes: $K=8$ with antithetic pairing yielded optimal results.
  • CFG: Using CFG during reward calculation degraded performance; it must be disabled for scoring.
  • Online vs. Offline: Online calculation (using the training model) outperformed offline (using a fixed reference).
  • Training Collapse: Restricting training to the suffix of timesteps ($\rho=0.6$) was critical to prevent the model from learning to generate trivial, textureless images.

Qualitative Results: Visual comparisons show that SOLACE-trained models produce images with better object counts, spatial relationships, and legible text compared to baselines, even when trained without external rewards.

5. Significance and Impact

• Scalability: SOLACE offers a scalable path to aligning T2I models without the bottleneck of collecting massive human preference datasets or training expensive reward models.
• Robustness: By using an intrinsic signal, the method reduces the risk of reward hacking associated with narrow external critics.
• Generalizability: The approach is architecture-agnostic (demonstrated on both SD3.5 and FLUX.1) and can be combined with external rewards to create a "best of both worlds" scenario where external rewards guide specific goals while intrinsic rewards maintain general quality and alignment.
• Future Directions: The paper suggests extending this to video and 3D generation (temporal consistency) and disentangling intrinsic signals for more precise task-targeted reward shaping.

In summary, SOLACE represents a paradigm shift in T2I post-training, proving that a generative model can effectively critique and improve its own outputs through self-confidence estimation, leading to more robust, aligned, and high-quality image generation.