💻 computer science

PaCo-RL: Advancing Reinforcement Learning for Consistent Image Generation with Pairwise Reward Modeling

This paper introduces PaCo-RL, a reinforcement learning framework that advances consistent image generation by combining a specialized pairwise consistency reward model (PaCo-Reward) with an efficient, cost-reduced optimization algorithm (PaCo-GRPO) to achieve state-of-the-art performance in preserving visual coherence across multiple images.

Original authors: Bowen Ping, Chengyou Jia, Minnan Luo, Changliang Xia, Xin Shen, Zhuohang Dang, Hangwei Qian

Published 2026-03-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Bowen Ping, Chengyou Jia, Minnan Luo, Changliang Xia, Xin Shen, Zhuohang Dang, Hangwei Qian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a director trying to film a movie. You need the main character to look exactly the same in every scene, wearing the same clothes, with the same face, even when the background changes from a forest to a spaceship.

In the world of AI image generation, this is incredibly hard. Current AI models are like talented but forgetful actors. They can draw a beautiful picture of a "red-haired wizard," but if you ask for a second picture of the same wizard, the AI might give him blue hair, a different beard, or make him look like a completely different person. This is the problem of Consistency.

The paper you shared, PaCo-RL, is a new toolkit designed to teach AI how to be a consistent actor. It solves this problem using two main "superpowers": a Smart Critic and a Smart Coach.

Here is how it works, explained simply:

1. The Problem: The "Forgetful" Artist

Current AI models are trained on massive libraries of pictures. They learn to draw what a prompt says (e.g., "a cat"), but they don't naturally understand continuity. If you ask for a set of images showing a cat eating, sleeping, and playing, the AI might draw three different cats.

Traditional methods try to fix this by showing the AI thousands of examples of "good" consistent sets. But creating those examples is expensive, slow, and hard to do perfectly.

2. The Solution: PaCo-RL (The Two-Part System)

The authors built a system called PaCo-RL (Pairwise Consistency Reinforcement Learning). Think of it as a two-step process to train the AI.

Part A: The Smart Critic (PaCo-Reward)

Before the AI can learn to be consistent, it needs a teacher who knows what "consistent" actually looks like.

The Old Way: Previous teachers (Reward Models) were like art critics who only judged if a picture was "pretty" or if it matched the text description. They didn't care if the character looked the same in the next picture.
The New Way (PaCo-Reward): This is a specialized critic trained to play a game of "Spot the Difference."
- Instead of looking at one picture, it looks at two pictures side-by-side.
- It asks: "Do these two images feature the same character in the same style?"
- It was trained on a massive dataset of thousands of these "pairs," where humans ranked them.
- The Analogy: Imagine a casting director who doesn't just look at a single audition tape. Instead, they look at two tapes side-by-side and say, "Yes, that is definitely the same actor," or "No, that's a different person." This critic is so good at spotting consistency that it outperforms all previous methods.

Part B: The Smart Coach (PaCo-GRPO)

Once we have a great critic, we need a way to train the AI artist using that critic's feedback. This is where Reinforcement Learning (RL) comes in. The AI tries, gets a score from the Critic, and tries again to get a higher score.

However, training AI to generate multiple consistent images is computationally expensive (it takes a lot of computer power and time). The authors introduced two tricks to make this faster and safer:

The "Sketch First" Strategy (Resolution-Decoupled Training):
- The Problem: Training on high-definition, full-size images is like trying to learn to paint a masterpiece by only using giant, heavy brushes. It's slow and exhausting.
- The Fix: The AI learns to paint small, low-resolution sketches during training. It learns the concept of consistency (keeping the face the same) on small images.
- The Result: Once it masters the concept on small images, it can instantly apply those skills to generate huge, high-definition images. It's like learning to ride a bike on a tricycle before getting on a full-sized one. This saves about 50% of the computing time.
The "Balanced Diet" Strategy (Log-Tamed Aggregation):
- The Problem: The AI has to balance two goals: "Make the images consistent" AND "Make sure the images match the text prompt." Sometimes, the AI gets obsessed with one goal and ignores the other. If it focuses too much on consistency, it might just generate the exact same image 10 times (boring!). If it focuses too much on the text, the characters change (confusing!).
- The Fix: The authors created a mathematical "volume knob" that prevents the AI from turning the volume up too high on just one reward. It keeps the "Consistency" and "Text Matching" rewards in a healthy balance, ensuring the AI doesn't go crazy with one goal and neglect the other.

3. The Results: What Happens?

When they tested this system:

Better Stories: The AI could now generate a whole set of images for a story where the characters looked exactly the same in every panel.
Better Editing: If you asked the AI to "make the woman laugh" in a photo, it kept her face and style exactly the same, only changing the expression.
Faster Training: Because of the "Sketch First" strategy, they trained the model twice as fast without losing quality.

Summary Analogy

Imagine you are teaching a robot to draw a comic book.

Old Method: You show the robot 1,000 examples of comic books and hope it figures out the pattern. It's slow and the robot often draws different characters for the same hero.
PaCo-RL Method:
- You give the robot a Super-Sharp Eye (PaCo-Reward) that can instantly tell if two drawings are of the same person.
- You let the robot practice on tiny napkin sketches (Resolution-Decoupled) to learn the rules quickly.
- You put a Safety Guardrail (Log-Tamed Aggregation) to make sure the robot doesn't get obsessed with drawing the same face over and over, but still keeps the character consistent.

The Bottom Line: PaCo-RL is a breakthrough because it teaches AI to be a consistent storyteller, not just a random picture generator, and it does so much faster and cheaper than before. This opens the door for AI to help write graphic novels, design video game characters, and create marketing campaigns where the brand looks the same everywhere.

1. Problem Statement

Consistent Image Generation aims to produce multiple images that share coherent identities, styles, and logical contexts (e.g., storytelling, character design, or image editing). While supervised learning has advanced image generation, it struggles with consistency due to:

Data Scarcity: A lack of large-scale datasets capturing visual consistency across multiple images.
Modeling Complexity: The difficulty of encoding subjective human perceptual preferences regarding consistency into standard supervised loss functions.
RL Limitations: Existing Reinforcement Learning (RL) approaches for image generation lack suitable reward models for consistency and efficient algorithms to handle the high computational cost of generating multiple high-resolution images.

2. Methodology: PaCo-RL Framework

The authors propose PaCo-RL, a comprehensive framework consisting of two core components: a specialized reward model (PaCo-Reward) and an efficient RL algorithm (PaCo-GRPO).

A. PaCo-Reward: Pairwise Consistency Reward Model

To address the lack of consistency-specific reward signals, the authors introduce a novel data synthesis and modeling pipeline:

PaCo-Dataset Construction:
- Automated Synthesis: They generate 2,000 prompts and use FLUX.1-dev to create image grids. A "sub-figure combinatorial pairing" strategy creates 33,984 unique ranking instances from 708 prompts.
- Human Annotation: Annotators rank images based on consistency with a reference. The dataset includes 54,624 annotated image pairs (consistent vs. inconsistent) augmented with Chain-of-Thought (CoT) reasoning.
- Scope: Covers 6 major categories (e.g., Character, Design Style, Story) and 32 subcategories.
Model Architecture & Training:
- Generative Formulation: Instead of using a regression head (which mismatches Vision-Language Models' nature), PaCo-Reward reformulates consistency evaluation as a generative task. It predicts the probability of generating a "Yes" token to indicate consistency between two images.
- CoT Integration: The model generates a "Yes/No" decision followed by a CoT reasoning sequence.
- Weighted Loss: A weighted likelihood objective balances the decision token ( $\alpha$ ) and the reasoning tokens. The optimal $\alpha$ was found to be 0.1, ensuring the model prioritizes the binary decision while using reasoning for interpretability and robustness.

B. PaCo-GRPO: Efficient RL Algorithm

To overcome the computational bottlenecks of training on multiple high-resolution images, PaCo-GRPO introduces two key strategies based on Group Relative Policy Optimization (GRPO):

Resolution-Decoupled Training:
- Strategy: The model trains on low-resolution images (e.g., $512 \times 512$ ) to compute rewards and update policies, while maintaining full-resolution generation ( $1024 \times 1024$ ) during inference.
- Benefit: This reduces computational cost quadratically (scaling with image size in Transformers) without sacrificing final output quality, as low-resolution samples still provide reliable reward signals.
Log-Tamed Multi-Reward Aggregation:
- Problem: Standard weighted sum aggregation often leads to reward domination, where one signal (e.g., consistency) overwhelms others (e.g., prompt alignment), causing training instability or degenerate outputs.
- Solution: The authors apply a logarithmic transformation to rewards with high variance (high coefficient of variation).
- Mechanism: If a reward's fluctuation exceeds a threshold $\delta$ , it is compressed via $\log(1 + R)$ . This prevents any single reward from dominating the optimization while preserving the relative ranking of samples.

3. Key Contributions

PaCo-Reward: A state-of-the-art pairwise reward model that outperforms existing baselines (including GPT-4/5 and specialized editing models) in aligning with human perceptions of visual consistency. It uniquely leverages a generative, CoT-enhanced approach.
PaCo-GRPO: An efficient RL algorithm that decouples training resolution from inference resolution and introduces a log-tamed aggregation mechanism to stabilize multi-objective optimization.
PaCo-Dataset: A large-scale, human-annotated dataset covering diverse consistency patterns (identity, style, logic) constructed via automated sub-figure pairing.
Unified Framework: Successfully unifies two distinct tasks—Text-to-ImageSet Generation (creating coherent sets from one prompt) and Image Editing (modifying attributes while preserving identity)—under a single RL framework.

4. Experimental Results

The framework was evaluated on T2IS-Bench (Text-to-ImageSet) and GEdit-Bench (Image Editing).

Reward Model Performance:
- On ConsistencyRank, PaCo-Reward-7B achieved a 10.5% gain in Accuracy and a 0.150 increase in Spearman's $\rho$ over the base Qwen2.5-VL-7B, outperforming all open-source and approaching proprietary models (GPT-5).
- On EditReward-Bench, it surpassed all open-source baselines and matched GPT-5 performance.
RL Generation Performance:
- Text-to-ImageSet: PaCo-RL achieved SOTA consistency metrics (10.3%–11.7% improvement) while preserving prompt fidelity, narrowing the gap between open-source (FLUX.1-dev) and closed-source models (GPT-4).
- Image Editing: Consistently improved Semantic Consistency (SC) and Prompt Quality (PQ) across base models (FLUX.1-Kontext, Qwen-Image-Edit) without degrading perceptual quality.
Efficiency & Stability:
- Training Speed: Resolution-decoupled training reduced training time by nearly 50% (e.g., 6.0h vs. 12.0h for full resolution) while achieving better convergence.
- Stability: The log-tamed aggregation kept the reward ratio between consistency and prompt alignment below 1.8, preventing the "reward domination" observed in baselines where consistency scores spiked but aesthetics collapsed.

5. Significance

Paradigm Shift: Demonstrates that Reinforcement Learning is a superior alternative to supervised learning for complex, subjective tasks like visual consistency, provided a specialized reward model and efficient algorithm are used.
Scalability: The resolution-decoupled strategy offers a practical pathway to scale RL for high-resolution, multi-image generation tasks, which were previously computationally prohibitive.
Generalization: The framework proves effective across diverse scenarios (storytelling, character design, editing), suggesting a unified solution for consistent image generation.
Open Science: The release of the PaCo-Dataset and the framework provides a new benchmark and toolset for the community to advance human-aligned image generation.

In conclusion, PaCo-RL effectively bridges the gap between human perceptual preferences and generative model optimization, offering a scalable, efficient, and high-performance solution for consistent image generation.