💻 computer science

SIVA-RL: Sensitivity-Invariance Visual Alignment for Multimodal Reinforcement Learning

SIVA-RL is a novel framework for multimodal reinforcement learning that enhances visual grounding by replacing operator-conditioned supervision with outcome-conditioned, sample-wise alignment, using a frozen audit policy to dynamically route interventions based on observed reward drops for improved reasoning performance across diverse benchmarks.

Original authors: Cheng Tang, Junzhi Ning, Min Cen, Wei Li, Xinyi Zeng, Pinxian Zeng, Rongbin Li, Qiming Zhu, Yuqiang Li, Junjun He, Yirong Chen, Ming Hu

Published 2026-07-16

📖 9 min read🧠 Deep dive

CC BY 4.0

Original authors: Cheng Tang, Junzhi Ning, Min Cen, Wei Li, Xinyi Zeng, Pinxian Zeng, Rongbin Li, Qiming Zhu, Yuqiang Li, Junjun He, Yirong Chen, Ming Hu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to solve a mystery. You show it a picture of a crime scene and ask, "Who did it?" If the robot guesses correctly, you give it a gold star. This is how modern "Reinforcement Learning" works for AI: it learns by trying, failing, and getting rewards for correct answers. But here's the catch: the robot might get the gold star for the wrong reasons. It might have memorized that "butlers" are usually the culprits in stories, or it might have guessed based on the text alone, completely ignoring the clues in the picture. It's like a student who gets an 'A' on a math test by guessing the answer key's pattern rather than actually understanding the numbers.

To fix this, scientists have tried to "poke" the robot's brain. They show it the same picture but with a piece of it covered up or scrambled. If the robot still gets the answer right, they assume it's not looking at the picture at all. If it gets it wrong, they assume it was looking. This is called "visual intervention." The idea is that by messing with the image, you can force the robot to pay attention to the visual evidence. However, this paper suggests that this old way of poking is a bit clumsy, like trying to teach someone to swim by throwing them into a pool and assuming they either sink or float based on a single splash.

Enter SIVA-RL, a new method that acts more like a smart coach than a clumsy pokes. The researchers found that the old "poke" method makes a big mistake: it assumes that every time you cover up a part of an image, the robot's reaction will be the same. But in reality, sometimes covering a piece of the image makes the robot fail miserably (because that piece was crucial), and sometimes it doesn't change the answer at all (because the robot was guessing anyway, or the covered part didn't matter). SIVA-RL stops guessing how the robot should react and instead watches how it actually reacts. It treats every single image as its own unique case, deciding exactly how to coach the robot based on the specific result of the "poke."

The Problem with "One-Size-Fits-All" Pokes

Imagine you are a teacher trying to help a student learn to identify different fruits. You have a picture of an apple. To test if the student is really looking at the apple, you cover up the stem.

The Old Way: You decide that any time you cover the stem, the student must get the answer wrong to prove they were looking. If they still guess "apple," you think they are cheating.
The Reality: Sometimes, covering the stem makes the student say, "I don't know!" (Great! They were looking at the stem). But other times, they still say "apple" because the red color and round shape were enough to guess correctly. Or maybe they were just guessing "apple" because it's a common fruit, and covering the stem didn't change their lazy guess.

The paper's authors, led by Cheng Tang and colleagues, discovered that the old method treats all these different reactions the same way. They call this "operator-conditioned" supervision, which is a fancy way of saying "we tell the robot what to do based on what we did to the image, not what the robot did with it."

The researchers ran a test where they took 906 examples where the robot got the answer right on the clean picture. They then covered up parts of the image in three different ways: making the whole image black, putting a random mask over it, or swapping small patches of the image (like a digital version of cutting out a piece of a puzzle and pasting it elsewhere).

The results were surprising. Even though they used the exact same "covering" tool, the robot's reaction was all over the place:

With the black image, the robot's score dropped 67.8% of the time, but stayed the same 32.2% of the time.
With random masks, it dropped 62.9% of the time, but stayed the same 37.1% of the time.
With patch swaps, it dropped 61.8% of the time, but stayed the same 38.2% of the time.

This means that the same "poke" can be a huge deal for one picture and a total waste of time for another. The old method didn't know the difference. It just assumed that if you covered the image, the robot should be sensitive to it.

The SIVA-RL Solution: A Smart Coach

The team proposed SIVA-RL (Sensitivity-Invariance Visual Alignment). Think of this as a coach who watches the student's reaction to the "poke" before deciding what to say next.

Here is how the SIVA-RL coach works in three simple steps:

1. The "PatchSwap" Poke
Instead of just blurring the whole image or covering it with a black box, SIVA-RL uses a technique called PatchSwap. Imagine you have a photo of a geometric shape. The coach takes a small square from one part of the photo (say, a corner of a triangle) and swaps it with a square from a far-away part of the same photo. This changes the local details but keeps the rest of the picture looking normal. It's like rearranging the furniture in a room; the room is still there, but the layout is slightly different.

2. The "Audit" (Checking the Reaction)
Before the robot learns anything, a "frozen audit" version of the robot (a snapshot of what it knew before the training started) looks at the original picture and the swapped picture. It tries to answer the question for both.

If the robot's score drops a lot (e.g., from 100% to 20%), it means the robot was really relying on that specific piece of the image. This is a Sensitivity signal.
If the robot's score stays the same (e.g., 100% to 98%), it means the robot either didn't need that piece, or it was just guessing. This is an Invariance signal.
If the score changes a little bit, it's ambiguous, and the coach might just ignore it for now.

3. The Tailored Lesson
This is where the magic happens. The coach doesn't use the same lesson for everyone.

For the "Sensitive" cases (Big Drop): The coach says, "Hey, you messed up when we moved that piece! That means you were looking at it, but maybe not carefully enough. Let's practice distinguishing that specific detail." The goal here is to make the robot more sensitive to the visual evidence.
For the "Invariant" cases (Small Drop): The coach says, "You got it right even when we moved that piece. That's good! But let's make sure you aren't just guessing. Let's practice staying consistent." The goal here is to make the robot stable and consistent, ensuring it isn't just relying on a lucky guess or a text shortcut.
For the "Ambiguous" cases: The coach says, "I'm not sure what's going on here, so let's skip this one for now."

The Results: Does It Work?

The researchers tested this new method on nine different benchmarks, which are like standardized tests for AI. These tests included math problems with diagrams, logic puzzles with pictures, and counting tasks. They used two different "backbones" (the underlying engines that drive the learning) called GRPO and DAPO, and they tested it on models of two sizes: 3 billion parameters (3B) and 7 billion parameters (7B).

The results were quite strong.

Overall Improvement: SIVA-RL improved the performance of the models in every single test. For the 7B models using the DAPO engine, the overall improvement was 14.98% relative to the baseline.
Vision-Dependent Tasks: The biggest wins were on tasks that really needed to look at the picture. For example, on vision-dependent reasoning tasks, the GRPO-7B model saw an 8.79 percentage-point gain. This suggests that SIVA-RL is particularly good at forcing the robot to actually look at the visual clues instead of just guessing.
Comparison to Others: The 7B SIVA-RL models performed better than many other famous models, including some that are much larger (like 72B parameter models) and even some proprietary models from big tech companies.

Why This Matters

The paper argues that we need to stop treating all "pokes" the same. Just because you cover up a part of an image doesn't mean the robot should fail. Sometimes it fails because it was paying attention; sometimes it fails because it was confused; sometimes it doesn't fail because it was guessing.

SIVA-RL suggests that the key to better AI reasoning is outcome-conditioned supervision. Instead of saying, "Because I covered the image, you must learn to be sensitive," the system says, "Because you actually dropped your score when I covered the image, let's learn from that specific drop."

The authors are careful to note that this doesn't prove the robot is now "thinking" like a human or that it has perfect visual grounding. It just means the robot is better at using the visual evidence it has. The method works by separating the act of creating the "poke" (the intervention) from the act of deciding what to teach based on the result (the supervision).

In the end, SIVA-RL is like a coach who stops shouting generic instructions and starts watching the player's specific mistakes. If the player trips on a specific rock, the coach helps them step over that rock. If the player trips because they were daydreaming, the coach helps them focus. By treating every mistake as a unique story, the robot learns to rely more on the picture in front of it, rather than just guessing the answer.

Technical Summary: SIVA-RL

Problem Statement

Reinforcement learning with verifiable rewards (RLVR) has advanced multimodal reasoning by optimizing task-level correctness without requiring dense reasoning trajectories. However, a critical limitation remains: a correct final answer does not guarantee that a vision-language model (VLM) grounded its prediction in visual evidence. Models may rely on language priors, answer-frequency biases, or memorized templates to achieve high rewards, particularly in vision-dependent tasks like counting or geometry.

Existing methods attempt to enforce visual dependence by contrasting model behavior on original images versus modified (intervened) images. These approaches typically employ operator-conditioned regularization, where the optimization objective is predetermined by the type of intervention (e.g., destructive interventions force separation, while mild transformations encourage consistency). The paper argues this assumption is flawed: identical visual operators produce heterogeneous outcomes across samples. A single intervention type can simultaneously cause a reward drop (indicating sensitivity to visual evidence) in some samples while leaving the outcome stable (indicating reliance on priors or redundant evidence) in others. Consequently, assigning a uniform supervision direction based solely on the intervention type is suboptimal.

Methodology: SIVA-RL

The authors propose SIVA-RL (Sensitivity–Invariance Visual Alignment), a framework that decouples intervention construction from supervision assignment. Instead of operator-conditioned regularization, SIVA-RL employs sample-wise, outcome-conditioned supervision.

1. Localized Intervention Construction

SIVA-RL constructs interventions using PatchSwap, a token-aligned, distance-constrained within-image patch swapping mechanism.

Mechanism: The image is partitioned into a grid aligned with visual tokens. Candidate cells are sampled, and valid replacement targets are selected from distant source cells (enforcing a minimum distance $d_{min}$ ).
Goal: This disrupts localized visual evidence and spatial relationships while preserving global image statistics and context, creating interventions that are harder than global masking but less destructive than complete removal.

2. Outcome-Conditioned Soft Routing

A frozen audit policy (a snapshot of the current model without gradients) evaluates each clean–intervention pair to determine the training signal.

Audit: The policy generates responses for both the clean image ( $x_{cl}$ ) and the intervened image ( $x_{ci}$ ). A verifier scores these responses to obtain rewards $R_{cl}$ and $R_{ci}$ .
Validity Gate: Pairs are only considered if the clean response is correct ( $R_{cl} \ge \tau_c$ ) and parseable.
Routing Weights: The observed reward drop $\Delta_i = R_{cl} - R_{ci}$ $Δ_{i} = R_{c l} - R_{c i}$ determines the supervision direction via soft routing weights:
- Sensitivity Alignment: Large reward drops indicate the model relied on the modified evidence. These pairs receive a weight to encourage separation (divergence) between the clean and intervened policy behaviors.
- Invariance Alignment: Small reward drops (where the answer remains stable) indicate the modification was irrelevant or the model relied on robust priors. These pairs receive a weight to encourage consistency (bounded alignment) with the clean response.
- Abstention: Ambiguous pairs (medium drops) are down-weighted or skipped.

3. Dual-Direction Alignment

The framework reuses the clean-rollout responses sampled for the base RL objective. These responses are re-scored under both visual conditions using the trainable policy.

Clean-Anchored Measure: The clean-conditioned likelihood acts as a stop-gradient anchor.
Loss Functions:
- $L_{sen}$ : Enforces a minimum divergence margin for high-drop pairs.
- $L_{inv}$ : Constrains the discrepancy below a maximum margin for low-drop pairs.
Total Objective: The final loss combines the base RL objective (GRPO or DAPO) with these auxiliary alignment terms: $L_{SIVA} = L_{base} + \lambda_{sen}L_{sen} + \lambda_{inv}L_{inv}$ .

Key Contributions

Diagnosis of Intervention Ambiguity: The paper demonstrates that the same visual intervention induces mixed outcome-drop and outcome-stable behaviors within clean-correct examples. It establishes that low-drop pairs are not necessarily reliable invariance signals, as they may stem from task-irrelevant perturbations or language shortcuts.
Outcome-Conditioned Selective Alignment: SIVA-RL formulates visual-intervention-based RL as a selective supervision problem. It routes pairs to sensitivity or invariance objectives based on the observed reward drop rather than the type of intervention, separating construction from supervision.
Scalable Implementation: The method is instantiated with token-aligned PatchSwap and compatible with both GRPO and DAPO backbones. It requires no additional stochastic sampling beyond the paired audit used for valuation.

Experimental Results

The authors evaluated SIVA-RL on 3B and 7B Qwen2.5-VL models across nine benchmarks, including mathematical, logical, counting, and vision-dependent tasks.

Performance Gains: SIVA-RL consistently improved all matched RL baselines.
- Overall: Relative improvements ranged from 6.83% to 14.98% across configurations.
- Vision-Dependent Tasks: The method yielded an 8.79 percentage-point gain on vision-dependent reasoning benchmarks, significantly outperforming gains on general multimodal reasoning.
- Comparison: The 7B SIVA-RL models achieved the strongest full-suite averages among reported 7B systems, outperforming several larger proprietary and general-purpose VLMs on specific math and vision tasks.
Ablation Studies:
- Intervention Construction: Distance-constrained PatchSwap outperformed random masking and unconstrained PatchSwap.
- Routing Strategy: Soft outcome-conditioned routing outperformed hard binary routing, confirming that the magnitude of the reward drop provides a smoother, more reliable supervision signal.
- Dual-Direction: Both sensitivity and invariance branches contributed to performance; the invariance branch provided a 1.85-point gain on vision-dependent subsets, proving that low-drop pairs offer useful consistency constraints.
Training Diagnostics: Analysis showed that both sensitivity and invariance routing remained active throughout training, with the model not collapsing to ignore interventions.

Significance and Claims

The paper claims that SIVA-RL shifts multimodal RL from operator-conditioned regularization to sample-wise, outcome-conditioned selective supervision. By acknowledging the heterogeneity of intervention effects at the sample level, the framework ensures that supervision is assigned based on the actual behavioral impact of the visual modification. This approach effectively forces models to ground predictions in visual evidence for tasks where it is required, while maintaining robustness for tasks where visual evidence is redundant or where priors are sufficient. The results suggest that this sample-wise valuation is a more effective alternative to fixed intervention definitions for improving visual reasoning in RLVR settings.