🤖 AI

Multimodal Reinforcement Learning with Adaptive Verifier for AI Agents

This paper introduces Argos, an adaptive agentic verifier that dynamically selects scoring functions to provide fine-grained, multi-dimensional rewards for multimodal reinforcement learning, thereby preventing reward hacking and achieving state-of-the-art performance in complex agentic tasks where standard outcome-based rewards and supervised fine-tuning fall short.

Original authors: Reuben Tan, Baolin Peng, Zhengyuan Yang, Hao Cheng, Oier Mees, Theodore Zhao, Andrea Tupini, Isar Meijer, Qianhui Wu, Yuncong Yang, Lars Liden, Yu Gu, Sheng Zhang, Xiaodong Liu, Lijuan Wang, Marc Poll

Published 2026-08-03

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Reuben Tan, Baolin Peng, Zhengyuan Yang, Hao Cheng, Oier Mees, Theodore Zhao, Andrea Tupini, Isar Meijer, Qianhui Wu, Yuncong Yang, Lars Liden, Yu Gu, Sheng Zhang, Xiaodong Liu, Lijuan Wang, Marc Pollefeys, Yong Jae Lee, Jianfeng Gao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to be a helpful assistant. You don't just want it to say the right words; you want it to actually see what's happening in the room, understand the space, and plan its actions carefully. This is the world of Multimodal Reinforcement Learning. Think of "multimodal" as the robot having both eyes (vision) and a brain (language) working together. "Reinforcement Learning" is like training a dog: if it does a trick right, it gets a treat (a reward); if it messes up, it gets nothing. For a long time, scientists have been trying to teach these AI agents to be smarter, but they hit a wall. The old way of training was like grading a student's essay only by checking the final answer. If the student guessed the right answer but got there by making up wild, fake facts, they still got an "A." This is bad for robots, because if a robot hallucinates (imagines things that aren't there) while planning to pick up a cup, it might knock over a vase.

To fix this, researchers realized they needed to check the reasoning steps, not just the final result. They needed a way to say, "Wait, you said the cup is on the table, but I don't see it there in the image." This paper introduces a new, clever system called Argos to solve exactly that problem. It's a bit like hiring a super-strict, multi-skilled teacher who doesn't just grade the final test score but also checks your homework, your logic, and whether you actually looked at the pictures you were supposed to.

The Problem: The "Fluent Liar"

Imagine you are playing a game of "Simon Says" with an AI. The AI is very good at talking. It can write long, confident sentences about how it sees a dog in a picture. But sometimes, it's just making things up to sound smart. In the old training methods, if the AI guessed the right answer at the end (e.g., "There are 4 dogs"), it got a reward, even if it hallucinated the dogs in its reasoning steps. The paper argues that this is dangerous. If an AI is going to control a robot arm or navigate a house, it can't afford to be a "fluent liar." It needs to be grounded in reality.

The Solution: The Adaptive Argos Verifier

The authors built a system called Argos (Adaptive Reward for Grounded & Objective Scoring). Think of Argos as a Swiss Army Knife for grading.

In the past, teachers used a single, blunt tool to grade every answer. If the answer was a number, they checked the math. If it was a yes/no, they checked the word. But Argos is smarter. It looks at the specific question and the AI's answer, then dynamically picks the right tool from a toolbox to grade it.

Here is how Argos works in plain English:

It reads the AI's "thought process." The AI doesn't just spit out an answer; it writes down its reasoning, often pointing to specific spots in an image (like "look at the red ball at coordinates x=50, y=100").
It chooses the right grader.
- If the AI points to a spot in a picture, Argos grabs a Visual Grounding Tool (like a magnifying glass) to check if that spot actually contains the object the AI described.
- If the AI is talking about a video and says "the person jumps at 5 seconds," Argos grabs a Video Checker to watch that specific clip and see if the jump actually happened.
- If the AI is doing math, it uses a Math Checker.
- If the AI is just writing a story, it uses a Logic Checker.
It gives a "Dense" Reward. Instead of just saying "Good job" or "Bad job" at the very end, Argos gives feedback on every step. If the AI correctly identified the ball but got the color wrong, it gets partial credit for seeing the ball. This helps the AI learn how to see, not just how to guess.

What They Found

The researchers tested this new Argos system on a bunch of different tasks, from simple picture quizzes to complex robot movements.

Beating the Hallucinations: The most exciting finding is that Argos significantly reduces "hallucinations." When the AI is forced to prove it saw what it was talking about, it stops making things up. In tests where the AI had to spot fake objects or tricky visual illusions, the Argos-trained models were much better at saying "I don't see that" instead of guessing.
Better at Planning: When asked to plan a sequence of actions (like "put the bowl on the table"), the Argos models were much more successful. They didn't just guess the steps; they actually reasoned through the spatial layout of the room.
Robotics Success: They even tested this on simulated robots. The robots trained with Argos were better at manipulating objects, like picking up a specific item or moving things around, compared to robots trained with the old methods.

What They Ruled Out

The paper makes a very important point about how to train these AI agents. They explicitly argue that just showing the AI good examples (Supervised Fine-Tuning) is not enough. You can show the AI a million perfect examples of how to reason, but if you don't actively check its work during the training process (the Reinforcement Learning phase), the AI will eventually adopt strategies that bypass the intended learning. It will learn to write fluent-sounding nonsense that happens to get the right answer by luck, rather than actually understanding the world. The paper suggests that without this active, adaptive checking system, the AI collapses back into making up facts.

How Sure Are They?

The authors are quite confident in their results because they tested the system on many different, real-world benchmarks (like BLINK for spatial reasoning and LIBERO for robotics). They didn't just simulate a simple game; they showed that their model outperformed other state-of-the-art models on these difficult tasks. However, they also note that this is a new approach, and while it works very well in their experiments, the field of AI is always evolving. They suggest that as the "teacher" models (the tools Argos uses to check answers) get better, Argos will get even better too.

In short, Argos is a smart, adaptive referee that ensures AI agents don't just talk the talk, but actually walk the walk by keeping their feet (and eyes) firmly on the ground. It turns the training process from a simple "guess the answer" game into a rigorous "show your work" exam, resulting in AI that is more reliable, less prone to lying, and better at interacting with the real world.

Technical Summary: Multimodal Reinforcement Learning with Adaptive Verifier for AI Agents

Problem Statement

Agentic reasoning models trained with Multimodal Reinforcement Learning (MMRL) have demonstrated increasing capabilities but are predominantly optimized using sparse, outcome-based rewards derived solely from final answers. This approach provides limited guidance on the quality of the intermediate reasoning process and often leads to visual hallucinations, where models generate fluent but ungrounded responses that lack consistency with visual evidence.

While existing methods have attempted to address this through the curation of grounded Supervised Fine-Tuning (SFT) annotations, empirical observations indicate that SFT alone is insufficient. During MMRL, models tend to collapse into ungrounded solutions when grounding signals are not explicitly verified, particularly under sparse reward regimes. Furthermore, computing richer, fine-grained rewards in MMRL is challenging because different samples may require distinct scoring functions, and teacher models often provide noisy reward signals.

Methodology: The Argos Framework

The authors introduce Argos (Adaptive Reward for Grounded & Objective Scoring), a principled verification framework designed to train multimodal reasoning models for agentic tasks. Argos reframes MMRL as a multi-objective optimization problem, dynamically selecting from a pool of teacher models and rule-based scoring functions to evaluate each training sample.

1. Adaptive Verifier Architecture

Argos operates as an LMM agent that parses a model's response (visual input $v$ , question $q$ , reasoning trace $r$ , and predicted answer $\hat{y}$ ) and adaptively composes a multi-objective reward process. It evaluates three key dimensions:

Final Response Accuracy: Computed via exact string matching, relative numerical accuracy (with 5% tolerance), or semantic equivalence via a language model.
Spatiotemporal Grounding:
- Spatial: For images, the system extracts 2D points mentioned in the reasoning trace. It uses an open-vocabulary object detection model (e.g., DINO) to generate pseudo ground-truth bounding boxes and a segmentation model (e.g., SAM-2) to refine masks. The score is binary, checking if the predicted point lies within the segmented object. For synthetic content (charts, maps), a pointing model is used.
- Temporal: For videos, the system extracts frame-level observations and segment-level events. Frame-level scores are computed similarly to spatial grounding. Segment-level events are evaluated by a powerful teacher reasoning model to verify visual-semantic accuracy against the corresponding video segment.
Reasoning Quality: A larger teacher model computes the conditional probability of the predicted answer given the question, reasoning trace, and visual input, measuring the logical consistency between the reasoning and the final answer.

2. Gated Aggregation

To prevent noisy intermediate rewards from biasing the model away from the correct outcome, Argos employs a gated aggregation function. If the outcome accuracy ( $R_{acc}$ ) falls below a threshold $\tau$ , the final reward is simply $R_{acc}$ . If $R_{acc} \ge \tau$ , the final reward is a weighted average of the outcome, grounding, and reasoning quality rewards:
$R_{final} = \frac{w_A R_{acc} + w_G R_{spatial} + w_R R_{reasoning}}{w_A + w_G + w_R}$

3. Data Curation Pipeline

The authors propose a pipeline to generate visually grounded reasoning traces for SFT cold-start:

Proposal Generation: Using a teacher model (GLM-4.1V) and object detection (Molmo-7B), the system extracts 2D points and temporal segments from visual inputs.
Trace Generation: The teacher model is prompted with overlaid images (showing coordinates) and timestamps to generate reasoning traces that explicitly reference these points and temporal segments.
Verification and Filtering: Argos scores the generated rollouts. Samples with low-quality generations (e.g., correct answers but poor grounding) are filtered out, ensuring the SFT dataset consists of high-quality, grounded examples.

4. Training

The model (based on Qwen2.5-VL 7B) undergoes SFT on the curated dataset, followed by MMRL using the GRPO algorithm. The advantage is computed based on the aggregated rewards from Argos.

Key Contributions

Argos Framework: The first principled verification framework with adaptive tool selection for MMRL, capable of dynamically routing different scoring functions (teacher models and rule-based) per sample to evaluate accuracy, grounding, and reasoning quality simultaneously.
Grounded Data Curation: A novel pipeline for generating reasoning traces visually grounded in space (2D points) and time (temporal segments), coupled with a filtering mechanism to remove low-quality rollouts.
Empirical Superiority: Demonstration that relying solely on SFT with grounded annotations is insufficient; the online adaptive verification during RL is critical to prevent reward hacking and collapse to ungrounded solutions.
State-of-the-Art Performance: The approach achieves SOTA results across diverse agentic benchmarks, including spatial reasoning, visual hallucination reduction, embodied task planning, and robotic control.

Experimental Results

The model was evaluated on multiple benchmarks under zero-shot settings:

Spatial Reasoning: On BLINK, Argos achieved 56.0% accuracy, outperforming the baseline Qwen2.5-VL (54.4%) and the larger Video-R1 (49.0%). On MindCube-t and CV-Bench, it also showed significant gains, particularly in 3D understanding.
Visual Hallucination: Argos significantly reduced hallucinations, achieving 85.3% on CounterCurate (vs. 61.4% for baseline) and 49.7% on HallusionBench. Notably, it outperformed Video-R1, which performed worse than the base model on HallusionBench.
Embodied AI: On the EB-Alfred benchmark, Argos improved task success rates by over 25% on "complex" tasks compared to the base model. On EB-Habitat, it outperformed CoT-prompted baselines by ~7% on complex visual tasks.
Robotics: On the LIBERO benchmark for continuous control, Argos achieved 84.2% average success, outperforming baselines including Video-R1 (83.5%) despite using significantly fewer training samples (85k vs. 270k).

Significance and Claims

The paper claims that Argos provides a robust solution to the "reward hacking" and "ungrounded reasoning" problems prevalent in current multimodal RL. By treating MMRL as a multi-objective optimization problem and using an adaptive verifier to aggregate noisy signals, the framework guides the policy toward $\delta$ -Pareto-optimal solutions, balancing final accuracy with intermediate reasoning quality and visual grounding.

The authors emphasize that their work demonstrates a crucial insight: curating grounded SFT data is necessary but insufficient. Without an online verification mechanism like Argos during the RL stage, agents inevitably revert to ungrounded solutions. The adaptive composition of scoring functions is shown to be effective in reducing hallucination and improving generalization to complex, long-horizon agentic tasks, including robotics and embodied navigation. The modular nature of Argos suggests it can scale with improvements in task-specific teacher models to further enhance multimodal agents.