Trust Your Critic: Robust Reward Modeling and Reinforcement Learning for Faithful Image Editing and Generation

1. Problem Statement

Reinforcement Learning (RL) has become a dominant paradigm for aligning text-to-image (T2I) generation and image editing models with human preferences. However, current RL pipelines suffer from a critical bottleneck: unreliable reward models (critics).

• Hallucinations & Noise: General-purpose Multimodal Large Language Models (MLLMs), when used as zero-shot reward models, frequently hallucinate, neglect objects, and lack precise spatial reasoning.
• Misguided Optimization: These noisy scores lead to "reward hacking," where models optimize for the wrong signals (e.g., preserving the original image to maximize consistency scores while ignoring editing instructions, or generating black shadows for short prompts to satisfy instruction-following metrics).
• Lack of Specialization: Existing methods often rely on generic MLLMs that fail to distinguish between fine-grained execution (did the edit happen?) and consistency (did the rest of the image stay the same?).

2. Methodology: The FIRM Framework

The authors propose FIRM (Faithful Image Reward Modeling), a comprehensive framework comprising specialized data curation pipelines, robust reward models, and novel reward formulations for RL.

A. Tailored Data Curation Pipelines

Instead of using raw MLLM scores, FIRM introduces distinct pipelines for editing and generation to synthesize high-quality reward data.

1. FIRM-Edit Pipeline ("Difference-First" Approach):

   • Insight: MLLMs struggle to directly judge if an edit is perfect but excel at identifying differences between two images.
   • Process:
     1. Difference Extraction: A state-of-the-art MLLM analyzes the source and edited images to generate a detailed textual description of visual differences (obvious and detailed changes).
     2. Scoring: This difference description, along with the original instruction and image pair, is fed to a robust MLLM evaluator.
   • Metrics: The evaluator scores two decoupled dimensions:
     • Execution: Accuracy of following the edit instruction.
     • Consistency: Preservation of unmodified regions.
   • Datasets: Constructed FIRM-Edit-370K (370k samples) by curating from open datasets and synthetically rewriting instructions to ensure a balanced distribution of low/high scores.

2. FIRM-Gen Pipeline ("Plan-Then-Score" Approach):

   • Insight: Direct holistic scoring leads to coarse evaluations. A structured breakdown mimics human cognitive processes.
   • Process:
     1. Planning: A powerful LLM (Qwen3-32B) acts as a "planner," extracting a checklist of fine-grained criteria (subjects, attributes, style, negative constraints) from the user prompt.
     2. Scoring: A strong MLLM (Qwen3-VL-235B) acts as an "evaluator," using the generated checklist to perform a step-by-step verification of the image against the prompt.
   • Datasets: Constructed FIRM-Gen-293K using diverse prompts and images generated by various models (Flux, SDXL, etc.) to prevent overfitting to specific generator artifacts.

B. Robust Reward Models

• Models: Two specialized 8B-parameter models are trained: FIRM-Edit-8B and FIRM-Gen-8B, initialized from Qwen3-VL-8B-Instruct.
• Benchmark: A new benchmark, FIRM-Bench, is created with 807 human-annotated samples (301 for editing execution, 256 for consistency, 250 for generation) to rigorously validate the critics.

C. Novel Reward Formulations for RL

To prevent reward hacking during the RL training of generative models, the authors propose a "Base-and-Bonus" strategy:

1. For Editing: Consistency-Modulated Execution (CME)

   • Problem: Simple linear weighting ($R = w_1 C + w_2 E$) causes models to output unchanged images to maximize Consistency ($C$) while ignoring Execution ($E$).
   • Solution: $R_{CME} = \text{Execution} \times (w_1 + w_2 \times \text{Consistency})$.
   • Effect: Execution becomes a necessary condition. If the model fails to edit (low Execution), the total reward is suppressed regardless of Consistency. Consistency acts as a shaping signal only after meaningful edits are made.

2. For Generation: Quality-Modulated Alignment (QMA)

   • Problem: Models exploit short prompts by generating trivial outputs (e.g., black shadows) that technically follow instructions but lack visual fidelity.
   • Solution: $R_{QMA} = \text{InsFollowing} \times (w_1 + w_2 \times \text{Quality})$.
   • Effect: Once instruction following is high, the reward is modulated by image quality, preventing the model from settling for low-fidelity solutions.

3. Key Contributions

1. FIRM Framework: A complete ecosystem for training robust, task-specific reward models, featuring novel "difference-first" (editing) and "plan-then-score" (generation) data pipelines.
2. High-Quality Datasets & Models: Release of FIRM-Edit-370K, FIRM-Gen-293K, and the corresponding FIRM-Edit-8B and FIRM-Gen-8B reward models.
3. FIRM-Bench: A comprehensive, human-annotated benchmark demonstrating that FIRM models achieve superior alignment with human judgment compared to proprietary (GPT-5, Gemini) and open-source (Qwen3-VL, InternVL) baselines.
4. Reward Formulations: Introduction of CME and QMA strategies that effectively mitigate reward hacking and balance competing objectives in RL.
5. SOTA Performance: The resulting models, FIRM-Qwen-Edit and FIRM-SD3.5, achieve state-of-the-art results in both editing and generation tasks.

4. Experimental Results

Benchmark Performance (FIRM-Bench)

• Editing: FIRM-Edit-8B achieved the lowest Mean Absolute Error (MAE) of 0.62 (Execution: 0.53, Consistency: 0.73), outperforming GPT-5 (0.67) and all open-source baselines.
• Generation: FIRM-Gen-8B achieved an MAE of 0.51, surpassing GPT-5 (0.52) and the largest open-source model Qwen3-VL-235B (0.56), despite having significantly fewer parameters.

RL Downstream Performance

• Image Editing (FIRM-Qwen-Edit):
  • Achieved a new SOTA score of 7.84 on GEditBench and 4.42 on ImgEdit.
  • Outperformed models trained with general-purpose Qwen3-VL-8B/32B reward models, which showed marginal gains or even performance drops due to noisy signals.
  • Required only 2,400 training samples to match the performance of models trained on 27k samples.
• Image Generation (FIRM-SD3.5):
  • Achieved 0.77 on GenEval, 87.16 on DPGBench, and 77.12 on TIIF.
  • Significantly outperformed heavily resourced models like BAGEL and OmniGen2, demonstrating that high-quality reward signals are more critical than sheer parameter scale.

Ablation Studies

• Reward curves confirmed that CME and QMA prevent the "collapse" seen in linear weighting strategies.
• General-purpose MLLMs (Qwen3-VL) were shown to assign artificially high scores to unchanged images (in editing) or low scores to complex generations (in generation), validating the need for specialized critics.

5. Significance

This work establishes that the quality of the critic is the limiting factor in RL-driven generative AI. By moving away from generic, hallucination-prone MLLMs toward specialized, pipeline-optimized reward models, FIRM demonstrates that:

1. Data Curation Strategy Matters: How reward data is synthesized (e.g., difference-first, checklist-based) is as important as the model architecture.
2. Reward Formulation is Critical: Mathematical formulations like CME and QMA are essential to align the optimization landscape with human intent, preventing degenerate solutions.
3. Efficiency: Specialized reward models can drive SOTA performance with significantly fewer training steps and data compared to using generalist models.

The paper sets a new standard for fidelity and instruction adherence in generative AI, with all datasets, models, and code made publicly available to foster further research in reward-guided alignment.