Technical Summary: AlphaGRPO

Problem Statement

Unified Multimodal Models (UMMs) have evolved from pure autoregressive architectures to hybrid AR-Diffusion models, capable of seamless visual understanding and generation. However, effectively reinforcing these models to leverage their intrinsic understanding for complex generation tasks—such as reasoning-based text-to-image (T2I) and self-reflective refinement—remains a significant challenge.

Current approaches often rely on a "cold-start" stage using proprietary models to synthesize high-quality data for Supervised Fine-Tuning (SFT) before applying Reinforcement Learning (RL). This paradigm risks attributing performance gains to the distillation of stronger teacher models rather than the activation of the target model's latent capabilities. Furthermore, existing RL methods for visual generation often utilize holistic scalar rewards (e.g., VIEScore) or fine-tuned reward models. These approaches suffer from poor discriminability in open-world scenarios, where they fail to distinguish nuanced semantic errors, leading to reward overfitting and unstable training signals.

Methodology: AlphaGRPO

The authors propose AlphaGRPO, a framework that applies Group Relative Policy Optimization (GRPO) directly to AR-Diffusion UMMs without a cold-start SFT stage. The method is built on three core components:

1. Unified Trajectory Formulation

AlphaGRPO conceptualizes multimodal generation as a single continuous trajectory $\tau = (y, z_{1 \to 0})$, where $y$ represents the autoregressive reasoning text tokens and $z$ represents the diffusion generation path. This formulation unifies two tasks:

• Reasoning T2I: The model generates reasoning text ($y$) to plan spatial layouts and extract world knowledge before generating the image ($z$).
• Self-Reflective Refinement: The model diagnoses misalignments in an initial generation and executes a correction strategy via a new reasoning trajectory.

The optimization objective employs GRPO to update both the reasoning policy and the generation flow simultaneously. A shared advantage $\hat{A}_i$, derived from the group reward of the final image, is propagated to update both components, ensuring that reasoning improvements directly correlate with visual quality.

2. Decompositional Verifiable Reward (DVReward)

To address the instability of scalar rewards, the authors introduce DVReward. Instead of asking a Multimodal Large Language Model (MLLM) to assign a holistic score (0–10), DVReward decomposes complex user requests into atomic, verifiable questions.

• Decomposition: An LLM breaks down the prompt into semantic questions (e.g., entity existence, spatial relationships) and quality questions (e.g., texture fidelity, geometry). Crucially, it converts abstract adjectives into observable physical phenomena (e.g., "Is there steam?" instead of "Is it hot?").
• Confidence Scoring: A pre-trained MLLM (Qwen3-VL) acts as a verifier. Instead of binary Yes/No outputs, it calculates the probability ratio of the "Yes" token ($P_{Yes} / (P_{Yes} + P_{No})$) to generate a continuous confidence score.
• Aggregation: The final reward is the geometric mean of the semantic and quality scores, providing a stable, interpretable, and discriminative signal.

3. False-Positive Rectification (FPR)

In self-reflective refinement, there is a risk that the model might "improve" a trajectory that was already degraded, creating false-positive optimization signals. AlphaGRPO introduces FPR, which enforces a validity constraint: if a refined trajectory does not strictly improve upon the initial input (i.e., $r(z_i) \leq r(z_{init})$), it is assigned the group's minimum reward. This ensures that ineffective refinement attempts result in negative advantages, suppressing model degradation.

Key Contributions

1. First GRPO Framework for AR-Diffusion UMMs: AlphaGRPO is the first framework to introduce GRPO training to AR-Diffusion Unified Models, unlocking latent reasoning and self-refinement primitives without requiring an additional cold-start stage.
2. DVReward Mechanism: The paper introduces a novel fine-grained reward mechanism that decomposes prompts into atomic verifiable questions. This approach provides stable supervision signals and demonstrates the correct methodology for using general MLLMs as reward models without compromising their generalist capabilities.
3. Generalization and Robustness: The framework validates that reinforcement learning can enhance both Reasoning T2I and Self-Reflective Refinement, achieving significant improvements across diverse benchmarks without explicit training on editing tasks.

Experimental Results

The authors evaluated AlphaGRPO on the Bagel model (a native UMM) across multiple benchmarks:

• Text-to-Image Generation: AlphaGRPO demonstrated consistent improvements on GenEval, TIIF-Bench, DPG-Bench, and WISE. Notably, training on self-reflective refinement alone yielded performance comparable to direct Reasoning T2I training, suggesting the acquisition of generalized reasoning capabilities.
• Image Editing (GEdit-Bench): Without explicit training on editing data, AlphaGRPO trained on self-reflective refinement achieved a score of 7.08 on GEdit-Bench, outperforming the Bagel baseline by 0.52. This indicates that the alignment learned via AlphaGRPO transfers effectively to editing tasks.
• Inference-Time Refinement: Applying Inference-time Self-Reflective Refinement (Inf. SRR) further boosted performance, achieving 83.9% on TIIF-Bench (Short) and outperforming Bagel by 5.8%.
• Ablation Studies:
  • DVReward consistently outperformed holistic scalar rewards (VIEScore) and human preference models (PickScore, HPSv3), particularly on long prompts and complex benchmarks.
  • Confidence Scoring (using token logits) proved superior to hard binary scoring, providing smoother gradients.
  • False-Positive Rectification was critical for preventing overfitting to noisy rewards, especially on challenging splits like TIIF-Bench Long.

Significance and Claims

The paper claims that AlphaGRPO effectively leverages the inherent understanding of unified models to guide high-fidelity generation. By eliminating the need for a cold-start stage and utilizing a decompositional reward mechanism, the authors demonstrate that RL can activate dormant reasoning and self-correction capabilities in AR-Diffusion models.

The significance lies in the shift from distilling knowledge from stronger teachers to enhancing the model's own intrinsic primitives. The results validate that a unified approach to reasoning and generation, guided by verifiable, atomic rewards, can achieve robust, generalized multimodal intelligence across both generation and editing tasks. The authors note that while the method is effective, it relies on the base model's ability to generate initial outputs and does not introduce new societal risks beyond those inherent in the base models.