Technical Summary: Guidance Contrastive Token Credit Assignment for Discrete Policy Optimization

Problem Statement

Recent advancements in reinforcement learning (RL) for generative models, particularly Group Relative Policy Optimization (GRPO) and its variants like DAPO, have demonstrated strong performance in domains ranging from mathematical reasoning to text-to-image synthesis. However, these methods rely on sample-level rewards, which are broadcast uniformly across all tokens in a generated sequence. This uniform credit assignment fails to capture fine-grained, token-level contributions.

In reality, not all tokens are equally important. In chain-of-thought (CoT) reasoning, specific tokens (e.g., mathematical derivations) drive the correctness of the answer, while others (e.g., punctuation, linking verbs) are fillers. Similarly, in text-to-image generation, certain image regions are directly associated with the text prompt, while others (backgrounds) are less relevant. Existing token credit assignment methods often rely on heuristics such as token confidence, gradients, or entropy, but they lack a unified mechanism to identify semantically critical regions across both visual and textual modalities.

Methodology: Guidance Contrastive Policy Optimization (GCPO)

The authors propose Guidance Contrastive Policy Optimization (GCPO), a novel algorithm that assigns per-token advantages by contrasting model predictions under positive and negative prompts. The method is inspired by Classifier-Free Guidance (CFG), a standard inference technique in text-to-image diffusion models.

1. Contrastive Guidance Mechanism

GCPO computes a per-token importance weight ($\eta$) by measuring the divergence between the model's prediction given a positive prompt ($x$) and a negative prompt ($x^-$).

• For Text-to-Image: The positive prompt is the user input, and the negative prompt is typically an empty string (unconditional generation).
• For Multimodal Reasoning (VLMs): Since instruction-tuned models do not natively support empty prompts, the authors construct a negative prompt by appending the suffix instruction "generate the wrong answer" to the original question. This creates a contrast between the model's implicit belief in a correct answer versus an incorrect one.

The importance weight is calculated using the Kullback-Leibler (KL) divergence between the token distributions under these two conditions:
$$\eta_{i,t} = D_{KL}(\pi_\theta(y_{i,t}|x, y_{i,<t}) \parallel \pi_\theta(y_{i,t}|x^-, y_{i,<t}))$$
Tokens with high $\eta$ values indicate regions where the model's prediction is highly sensitive to the prompt, signifying semantic importance.

2. Rank-Based Normalization

Raw KL divergence values can vary wildly in scale across different tasks, prompts, and sequence lengths. Standard normalization techniques like softmax or min-max are prone to concentrating probability mass on outliers (e.g., the first token). To address this, GCPO employs histogram equalization (rank-based normalization).

• The unnormalized $\eta$ values are converted to cumulative distribution function (CDF) values within the sequence.
• This ensures that the distribution of normalized weights is consistent across samples, regardless of the absolute scale of the divergence, facilitating stable training without task-specific hyperparameter tuning.

3. Policy Optimization Objective

The normalized weights ($\eta_{normalized}$) are used to scale the per-sample advantage ($\hat{A}_i$) to obtain a per-token advantage ($\hat{A}^{GCPO}_{i,t}$):
$$\hat{A}^{GCPO}_{i,t} = \eta_{normalized, i,t} \cdot \hat{A}_i$$
This per-token advantage replaces the uniform advantage in the standard GRPO objective, allowing the policy update to focus more heavily on semantically critical tokens.

Key Contributions

1. Novel Token Credit Assignment: GCPO introduces a method to derive per-token importance weights by contrasting positive and negative prompts, effectively identifying key reasoning steps or visual regions without requiring explicit token-level rewards.
2. Extension to Language and Multimodal Domains: While CFG is standard in image generation, the paper successfully extends this concept to discrete language generation and multimodal reasoning by designing a specific negative prompt strategy ("generate the wrong answer") that reveals the model's implicit correctness beliefs.
3. Robust Normalization: The proposal of rank-based normalization (histogram equalization) solves the instability issues associated with raw divergence metrics in RL training, ensuring smooth optimization across diverse tasks.
4. Generalizability: The method is shown to be effective across both text-to-image generation and complex multimodal reasoning tasks, outperforming existing baselines.

Experimental Results

The authors evaluated GCPO on two primary benchmarks:

Text-to-Image Generation

• Benchmark: GenEval.
• Base Model: Janus-Pro-7B.
• Results: GCPO achieved an overall score of 0.89, outperforming the base model (+0.09), the previous RL method Janus-Pro-R1, and the GRPO baseline (0.85). It matched the performance of significantly larger models (e.g., Qwen-Image-2507, BAGEL).
• Observation: The most significant gains were in "Counting" and "Color Attribution" tasks, where precise token-level alignment with the prompt is critical.

Multimodal Reasoning

• Benchmarks: MathVerse, MathVision, MM12K, LogicVista, and MMMU-Pro.
• Base Models: Qwen2.5-VL-7B and Qwen3-VL-8B.
• Comparison: GCPO was compared against GRPO, DAPO, and VPPO (a prior work focusing on visually dependent tokens).
• Results: GCPO consistently outperformed all baselines. For instance, on the Qwen3-VL-8B model, GCPO achieved 55.3% on MMMU-Pro, surpassing VPPO (52.4%) and DAPO (50.5%).
• Ablation: The study confirmed that KL divergence is the most effective metric compared to Information Gain or absolute difference, and that the "generate the wrong answer" negative prompt outperforms other strategies like blank images or generic prompts.

Significance and Claims

The paper claims that GCPO is a simple, general, and scalable optimization strategy for discrete policy learning. Its significance lies in:

• Refining Learning Signals: By moving from uniform sample-level rewards to fine-grained token-level advantages, GCPO enables models to focus on the "substance" of reasoning traces and relevant visual regions, rather than treating all tokens equally.
• Bridging Inference and Training: It successfully adapts an inference-time technique (CFG) into a training-time reward shaping mechanism, demonstrating that contrastive signals can guide policy optimization even in domains where CFG is not explicitly used during sampling.
• Superiority over Heuristics: Unlike previous token credit assignment methods that rely on static heuristics (confidence, gradients), GCPO derives weights dynamically based on the model's own sensitivity to prompt conditions, providing a more semantically grounded signal.

The authors note limitations, specifically that the method requires the model to have initial capabilities to differentiate correct from incorrect answers (to respond to the negative prompt) and is strictly applicable to prompt-dependent RL tasks. However, they assert that GCPO paves the way for future advancements in discrete policy optimization by providing a robust mechanism for token-level credit assignment.