Technical Summary: SRUM - Fine-Grained Self-Rewarding for Unified Multimodal Models

1. Problem Statement

Unified Multimodal Models (UMMs) have advanced significantly by integrating vision-language understanding and generation within a single framework. However, a fundamental paradox persists: the understanding capability of current UMMs often exceeds their generation capability. A model may correctly identify that a generated image fails to align with a complex prompt (demonstrating strong understanding) yet remain incapable of generating a faithful image from that same prompt.

This gap suggests that the "teacher" (understanding module) is more capable than the "student" (generation module). Existing post-training methods often rely on external reward models, human preference labels, or rule-based rewards, which are resource-intensive, require curated paired data, or lack universality across tasks. The paper posits that the solution lies within the model's own architecture: leveraging the internal understanding module to guide and improve the generation module without external supervision.

2. Methodology: SRUM Framework

The authors propose SRUM (Self-Rewarding for Unified Multimodal Models), a post-training framework that creates a synergistic feedback loop where the model evaluates and rewards its own generations. The process consists of three main stages:

A. Self-Rewarding Data Generation

The UMM utilizes its "think" mode (Chain-of-Thought) to generate candidate images based on text prompts. To facilitate fine-grained evaluation, the model (aided by external segmentation tools like SAM for some architectures, or native grounding for others) generates bounding boxes for relevant regions. Crucially, the external component serves only as a localization aid; the semantic scoring is performed entirely by the UMM's own understanding module.

B. Global-Local Dual Reward System

To address the insufficiency of holistic scores for detailed improvement, SRUM employs a two-tiered reward mechanism:

1. Local Reward: Evaluates object-level fidelity, attribute binding, and visual artifacts on a strict scale of $[-1.0, 1.0]$. It includes a mandatory "Reason" field to elicit interpretable rationales (similar to Chain-of-Thought), enhancing reliability. A non-linear penalty is applied to severe distortions to reflect human visual sensitivity.
2. Global Reward: Evaluates the overall composition, spatial alignment, and scene plausibility. For prompts lacking specific compositional directives, a neutral score range is applied to ensure fairness.

These scores are converted into a dense reward map ($R$) for local regions and a global scalar ($\alpha$) for the whole image. The understanding module remains frozen during this phase, acting solely as an evaluator.

C. Reward-Weighted Training

The core training objective translates the understanding module's insights into the generation module's parameters. The loss function ($\mathcal{L}_{Total}$) combines two terms:

1. Reward-Weighted Loss ($\mathcal{L}_r$): This term modulates the standard velocity prediction loss in flow-based frameworks. It weights the squared error between the predicted velocity and the target velocity by the product of the global and local rewards ($\alpha \cdot R$).
   • If feedback is positive ($\alpha \cdot R > 0$), the model is encouraged to preserve the current generation.
   • If feedback is negative ($\alpha \cdot R < 0$), the model is penalized to promote change.
2. Reference Constraint ($\mathcal{L}_{ref}$): A regularizer that penalizes the squared $\ell_2$ distance to the target velocity of the artifact-free latent. This prevents "reward hacking" and ensures training stability, a concept aligned with methods like Direct Preference Optimization (DPO).

The final objective is:
$$\mathcal{L}_{Total} = \mathcal{L}_r + \lambda_c \cdot \mathcal{L}_{ref}$$
where $\lambda_c$ is a hyperparameter (optimized at 0.5 in experiments).

3. Key Contributions

• First Comprehensive Self-Rewarding Framework for UMMs: The paper introduces the first post-training framework specifically designed to bridge the understanding-generation gap in UMMs using the model's internal capabilities, eliminating the need for external reward models or human labels.
• Global-Local Dual Reward Design: The authors propose a novel reward structure that combines high-level compositional assessment with fine-grained, object-level feedback. This multi-scale guidance is shown to be critical for complex generation tasks.
• Demonstrated Generalization: The framework is validated across different UMM architectures (Bagel, BLIP3o) and shows strong transferability in both in-domain (compositional) and out-of-domain (reasoning-based) settings.

4. Experimental Results

The authors evaluated SRUM on the T2I-CompBench and T2I-ReasonBench benchmarks, using QwenVL-2.5 as the primary evaluator.

• Performance Gains:
  • On T2I-CompBench, SRUM improved the overall score of the Bagel (CoT) baseline from 82.18 to 88.37.
  • On T2I-ReasonBench, the score increased from 43.82 to 46.75.
  • Significant improvements were observed in spatial reasoning, 3D spatial relationships, and complex compositional tasks.
• Ablation Studies: Removing either the local or global reward resulted in significant performance drops, confirming the necessity of both scales. Removing the reference constraint also degraded performance, highlighting its role in stability.
• Comparison with External Rewards: SRUM outperformed methods using external VLMs (e.g., QwenVL-7B/32B) as reward models. The internal scorer demonstrated better calibration and separation between accepted and rejected samples, suggesting that intrinsic multi-scale design is more effective than simply scaling up external parameters.
• Efficiency: The self-rewarding process is computationally efficient, requiring fewer than 4 GPU-hours to score 6,000 candidate images on an NVIDIA H100, with negligible overhead compared to standard Supervised Fine-Tuning (SFT).
• Preservation of Understanding: The method did not degrade the model's core understanding capabilities on benchmarks like MME, MM-Vet, and MMMU, and in some cases (MMVP), performance slightly improved.

5. Significance and Claims

The paper claims that SRUM establishes a powerful paradigm for enabling UMMs to achieve self-improvement. By treating the understanding module as an internal teacher, the framework successfully bridges the gap between comprehension and generation.

The authors emphasize that this approach:

• Decouples improvement from external data: It removes the dependency on laborious human labeling or external reward models.
• Enables multi-scale refinement: The dual reward system addresses the specific failure modes of traditional T2I models regarding spatial relationships and attribute binding.
• Offers a scalable path: The method is applicable to various UMM designs and demonstrates robust generalization, suggesting a viable path for future research into self-correcting multimodal systems.

The work concludes that while SRUM is a preliminary exploration, it validates the principle of self-reward as a promising direction for the synergistic development of understanding and generation within a single model. Future work may involve refining the prompting strategies for the scoring module and scaling the method to larger datasets.