Technical Summary: Scaling Multi-Reference Image Generation with Dynamic Reward Optimization

Problem Statement

While personalized image generation has advanced significantly, Multi-Reference Image Generation (MRIG) remains a challenging task. MRIG requires the effective integration of multiple reference images of diverse types (e.g., subject, pose, style, lighting, background) into a single coherent output. Existing research is hindered by two primary limitations:

1. Insufficient Evaluation: Current benchmarks typically focus on simple tasks with limited numbers of reference images or single reference types. They lack the complexity to evaluate scenarios requiring the integration of numerous and diverse references, and they often rely on coarse-grained metrics that fail to assess performance across specific reference types.
2. Performance Degradation in Open-Source Models: Mainstream open-source models struggle in complex MRIG scenarios. While they perform adequately with a few references, their performance deteriorates rapidly as the number of mixed-type reference images increases, creating a significant gap compared to state-of-the-art (SOTA) closed-source models.

Methodology: DyRef Framework

To address these challenges, the authors propose DyRef, a two-stage training framework designed to scale MRIG capabilities through dynamic reward optimization.

Stage I: Supervised Fine-Tuning (SFT)

The first stage equips the model with basic capabilities to handle complex MRIG tasks.

• Baseline: The authors utilize Qwen-Image-Edit-2511 (a dual-stream MMDiT backbone) as the training baseline.
• Data: Approximately 14,000 training samples are constructed using a pipeline that generates target images via Text-to-Image (T2I) models and derives corresponding reference images (subject, background, style, lighting, pose) using segmentation and editing models.
• Training: The model is fine-tuned using LoRA with Flow Matching loss to learn the fundamental mapping between complex multi-reference inputs and target outputs.

Stage II: Dynamic Reward Optimization

The second stage employs Reinforcement Learning (specifically Flow-GRPO) to align the model with human preferences and improve performance on difficult samples. This stage introduces two novel components:

1. Difficulty-aware Advantage Reweighting (DAR):

   • Motivation: Standard RL training tends to be dominated by simpler samples (fewer references), while complex samples (many mixed-type references) receive insufficient learning signals due to lower initial reward scores.
   • Mechanism: DAR dynamically adjusts the optimization objective by assigning group weights based on the average reward score of samples within a group. It increases the weight of underperforming groups (those with numerous mixed-type references) to encourage the model to focus on these difficult cases, while preventing overfitting to well-performing simple samples.

2. Discriminative Reward Scaling (DRS):

   • Motivation: Reward signals from standard visual encoders (e.g., CLIP, SigLIPv2) often exhibit a narrow distribution, where high-quality and low-quality samples receive similar scores, weakening gradient signals for policy updates.
   • Mechanism: DRS applies a transformation function to the raw reward scores to enlarge the intra-group reward differences. This amplification ensures that superior samples receive significantly higher rewards than inferior ones, facilitating more effective policy optimization.

Reward Signals

The framework utilizes a hybrid reward system:

• Style Alignment: Uses CSD (a style similarity metric) to guide style-related tasks.
• Semantic Consistency: Uses SigLIPv2 (or CLIP) to measure the semantic consistency between generated images and target images.

Key Contributions

1. OmniRef-Bench: The authors introduce a new benchmark comprising 395 expert-annotated instances covering five reference types (subject, background, style, lighting, pose) and up to four simultaneous reference types. It features a hybrid evaluation strategy combining objective quantitative metrics and Multimodal Large Language Model (MLLM)-based assessments with fine-grained criteria for each reference type.
2. DyRef Framework: A two-stage training approach that successfully bridges the performance gap between open-source and closed-source models in complex MRIG tasks by integrating SFT with DAR and DRS.
3. Empirical Validation: Extensive experiments demonstrate that DyRef significantly improves open-source models on OmniRef-Bench and generalizes well to single-image editing benchmarks.

Experimental Results

• OmniRef-Bench Performance: DyRef significantly outperforms existing open-source models (e.g., OmniGen2, DreamOmni2, BAGEL) by at least 12% in objective metrics and 29% in MLLM evaluations. Notably, it achieves results comparable to SOTA closed-source models like Nano Banana Pro and Seedream 4.5, particularly excelling in style and pose-related metrics.
• Generalization: The method improves performance on single-image editing benchmarks (ImgEdit and DreamBench++), indicating that the enhancements in multi-reference capabilities do not compromise basic image editing skills.
• Robustness: The approach shows effectiveness across different model scales (tested on 9B and 20B parameter models) and maintains performance on unseen reference types.
• User Study: A study with human experts confirms that the MLLM-based evaluation aligns well with human preferences (Pearson correlation of 0.9), and DyRef-generated images rank highly against closed-source baselines.

Significance and Claims

The paper claims that the primary contribution of this work is addressing the critical performance degradation observed in open-source models as reference complexity and counts increase. By introducing OmniRef-Bench, the authors provide a necessary tool to rigorously evaluate complex MRIG scenarios, revealing the limitations of current methods. Through DyRef, they demonstrate that dynamic reward optimization (specifically DAR and DRS) can effectively mitigate performance variance across varying reference scales.

The authors position their work as a step toward making high-fidelity, complex multi-reference image generation accessible via open-source models, which is crucial for applications in professional visual design and advertising where coherent integration of varied visual elements is essential. They acknowledge that while DyRef narrows the gap significantly, complex MRIG remains an open problem, particularly regarding instruction following and feature disentanglement in extreme scenarios.