🤖 AI

Improving Generalization Robustness of Multimodal RLVR

This paper proposes Prompt-Invariant RLVR (PIRL), a robust post-training method that combines a dynamic trinary reward and an embedding-space consistency regularizer to mitigate the brittleness of standard RLVR in multimodal models, thereby significantly improving generalization across semantically equivalent but perturbed prompts.

Original authors: Pengfei Zhou, Zhiwei Tang, Xiaopeng Peng, Chenrui Zhou, Lama Moukheiber, Yixing Ma, Bin Xu, Jiajun Song, Zhenglin Wan, Wangbo Zhao, Jiasheng Tang, Bohan Zhuang, Fan Wang, Yang You

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Pengfei Zhou, Zhiwei Tang, Xiaopeng Peng, Chenrui Zhou, Lama Moukheiber, Yixing Ma, Bin Xu, Jiajun Song, Zhenglin Wan, Wangbo Zhao, Jiasheng Tang, Bohan Zhuang, Fan Wang, Yang You

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a brilliant but literal-minded robot how to solve puzzles. You show it a picture of a cat and ask, "What animal is this?" The robot learns to say "Cat" perfectly. But then, you change the question slightly to, "Identify the feline in the image." Suddenly, the robot freezes or gives a wrong answer. This is the world of Multimodal Large Language Models (MLLMs)—super-smart AI systems that can see pictures and read text at the same time. To make these robots even smarter, scientists use a technique called Reinforcement Learning with Verifiable Rewards (RLVR). Think of this like a video game where the robot gets a "point" only if its answer matches a strict checklist (like a specific format or a correct number). The goal is to train the robot to reason better and get more points. But here's the catch: if the robot learns to game the checklist rather than truly understanding the puzzle, it becomes fragile. It might ace the test when the questions look exactly like the practice ones, but fail miserably if you just rephrase the question. This is a huge problem for real-world uses, like medical diagnosis or legal advice, where a robot that breaks when you change the wording could be dangerous.

Enter a new study that tries to fix this "brittleness." The researchers, led by Pengfei Zhou and colleagues, discovered that standard training methods are teaching AI to be "format snobs." They found that when an AI gets a reward for a correct answer, the reward system often mixes up two things: getting the answer right and following the formatting rules (like putting the answer inside a specific box). If the AI forgets to put the answer in the box, it gets zero points, even if the answer is perfect. This confuses the AI, making it focus on the box instead of the math or logic. Furthermore, the AI only practices on a tiny, specific set of question styles. When it faces a new way of asking the same question in the real world, it panics.

To solve this, the team proposed a new training method called Prompt-Invariant RLVR (PIRL). They introduced two main tricks. First, they gave the AI a "Dynamic Trinary Reward." Instead of just a simple "Right/Wrong" score, the AI now gets a three-tiered score: +1 for a correct answer in the right format, 0 for a correct answer in the wrong format, and -1 for a format failure. This teaches the AI that the content of the answer matters even if the formatting is messy. Second, they added a "stunt double" to the training. They created an adversarial agent that constantly rewrites the questions in subtle ways (like changing the wording or the instruction style) while keeping the meaning the same. The AI is then forced to give the same correct answer no matter how the question is dressed up. It's like training a student not just to solve a math problem on a specific worksheet, but to solve it whether it's written in cursive, printed, or spoken aloud.

The results are promising. When they tested their new method against the old standard (called GRPO), they found that the old method's performance dropped significantly when the questions were rephrased (a "stress test"). For example, on some math benchmarks, the standard method's accuracy fell by about 3% to 4% when the prompt changed. In contrast, the new PIRL method stayed remarkably steady, with its accuracy dropping by only 1% or less. In high-stakes areas like medical and legal questions, where reliability is key, PIRL maintained its accuracy much better than the competition. The study suggests that by separating the reward for "being right" from the reward for "looking right," and by practicing on a wider variety of question styles, we can build AI that is not just smart, but also robust and reliable enough to handle the messy, unpredictable way humans actually talk.

Technical Summary: Improving Generalization Robustness of Multimodal RLVR

1. Problem Statement

The paper addresses the prompt-robustness gap in Multimodal Large Language Models (MLLMs) fine-tuned via Reinforcement Learning with Verifiable Rewards (RLVR). While RLVR significantly improves accuracy on standard benchmarks by optimizing against deterministic signals (e.g., regex matching or symbolic verification), these gains are brittle. The authors observe that simply paraphrasing a question or altering the prompt template can cause a sharp degradation in performance, rendering models unreliable for high-stakes deployment (e.g., medical or legal VQA).

The authors identify two root causes within the standard RLVR objective:

Reward Entanglement: Standard binary verifiers (e.g., $V = V_f \cdot V_c$ ) conflate format compliance ( $V_f$ ) with semantic correctness ( $V_c$ ). A zero reward cannot distinguish between a model that failed to understand the task and one that merely failed to follow the output format. This leads to a loss of informative reward variance, particularly when format compliance is high, causing the gradient signal to become biased toward template-specific behaviors rather than general reasoning.
Prompt Dependency: Standard RLVR optimizes over a fixed training distribution ( $P_{train}$ ). Theoretical analysis (Proposition B.5) demonstrates that this objective is non-identifiable: policies that perform perfectly on the training templates but fail on unseen, semantically equivalent prompts can achieve the same training reward. Consequently, the model learns to exploit specific prompt templates rather than learning prompt-invariant reasoning.

2. Methodology: Prompt-Invariant RLVR (PIRL)

To address these issues, the authors propose Prompt-Invariant RLVR (PIRL), which operationalizes two design conditions: Reward Decomposition (C1) and Invariance Control (C2).

2.1. Dynamic Trinary Reward (DTR)

To satisfy Reward Decomposition, PIRL replaces the standard binary verifier with a Dynamic Trinary Reward:
$R = R_t + \alpha_{rew} R_s$

$R_t$ (Format-Aware Trinary Verifier): Assigns $+1$ for a correct answer in the correct format, $0$ for a wrong answer in the correct format, and $-1$ for a format failure. This decouples format compliance from semantic correctness.
$R_s$ (Semantic Score): A bounded semantic score (derived from an NLI scorer) that provides variance within format buckets. Even if the format is correct, $R_s$ can differentiate between degrees of semantic correctness.
Decoupled Advantage Normalization (DAN): To prevent the scaling of the semantic score from dominating the trinary signal during group normalization, the authors propose z-scoring $R_t$ and $R_s$ separately within each rollout group before combining them.

2.2. Embedding-Space Adversary and Consistency Regularization

To satisfy Invariance Control, PIRL introduces an adversarial training loop:

Adversarial Agent: An adversary perturbs the instruction token embeddings within a bounded $\ell_2$ ball ( $\|\delta\|_2 \le \epsilon_{emb}$ ). This simulates worst-case rewording of instructions without altering the visual input or the ground-truth answer. The adversary aims to minimize the policy's expected DTR reward.
Consistency Regularization: To ensure the policy does not collapse under these perturbations, a consistency loss is added. This loss minimizes the Kullback-Leibler (KL) divergence between the token distributions of the policy under the clean prompt and the adversarial (perturbed) prompt. This acts as a tractable surrogate for enforcing answer-level invariance across the semantic equivalence class.

2.3. Training Objective

The overall objective combines the clipped GRPO loss (using DAN-normalized advantages from DTR) with the consistency penalty:
$\min_{\theta} \mathbb{E}_{\mathbf{x} \sim \mathcal{D}} \left[ \mathcal{L}^{DAN}_{G}(\pi_\theta, \bar{q}; R_t, R_s) + \lambda \mathcal{L}_{cons}(\theta; \mathbf{x}, \bar{\phi}) \right]$
where $\bar{q}$ represents the frozen adversarial samples generated via projected ascent.

3. Key Contributions

Identification of the Prompt-Robustness Gap: The paper formally reveals that standard RLVR leads to overfitting on prompt formats, creating a gap where models degrade on paraphrased prompts despite high training accuracy.
Theoretical Formulation: The authors formalize prompt robustness as optimization over semantic equivalence classes. They prove that standard binary rewards cause a reward-variance bias and that the training objective suffers from non-identifiability regarding prompt-invariant policies.
PIRL Framework: They propose a concrete method (PIRL) that instantiates reward decomposition via DTR and invariance control via an embedding-space adversary and consistency regularization.
Empirical Validation: Extensive experiments across Exam, Medical, and Legal VQA benchmarks demonstrate that PIRL significantly reduces performance drops under stress testing compared to standard GRPO and multi-template baselines.

4. Experimental Results

The authors evaluated PIRL on Qwen2.5-VL-7B and Qwen3-VL-8B across three domains:

Template Stress Test (T-Stress): Models were tested on prompts with varying numbers of unseen templates ( $N=1$ $N = 1$ to $15$).
- Standard GRPO showed significant degradation (e.g., $\Delta T \approx -4.4\%$ on MathVista for Qwen2.5).
- PIRL maintained stability, often showing positive or negligible drops (e.g., $\Delta T \approx +0.3\%$ on MathVista).
- PIRL outperformed the Multi-Template (GRPO-MT) baseline, which only partially mitigated the issue.
Dynamic Evaluation (D-Test): Using operator-based mutations (paraphrasing, layout changes, etc.).
- PIRL consistently achieved the smallest mean performance drop across all benchmarks compared to GRPO and GRPO-MT.
- In high-stakes domains (Medical and Legal), PIRL preserved accuracy where GRPO degraded significantly (e.g., on PathVQA, GRPO dropped $-6.7\%$ while PIRL dropped only $-0.8\%$ ).
Ablation Studies: The full PIRL configuration (MT + ADV + DTR) yielded the best results. While the ablation did not definitively isolate the individual necessity of DTR or the adversary beyond the multi-template baseline in all metrics, the combination provided the most robust performance with no detectable cost.

5. Significance and Claims

The paper claims that PIRL restores the robustness of RLVR-tuned models to a level comparable to or better than the base model, effectively closing the prompt-robustness gap.

Reliability: The method is particularly significant for high-stakes domains (medical, legal) where a model's inability to handle reworded instructions renders it unreliable regardless of its raw accuracy.
Mechanism: The work demonstrates that the fragility of RLVR is not an inherent limitation of reinforcement learning but a consequence of reward entanglement and limited training coverage. By decoupling format from semantics and enforcing invariance, the model learns to generalize across the semantic equivalence class rather than memorizing templates.
Modesty: The authors acknowledge that robustness does not guarantee correctness (a stable model can still be confidently wrong) and that their invariance objective is a tractable relaxation of the full semantic equivalence class (operating in embedding space rather than discrete token space). They position PIRL as a method to support, not replace, expert human judgment in critical domains.