🤖 AI

Teaching MLLMs to Say No: Generalized Referring Expression Comprehension via Refusal Calibrated GRPO

This paper proposes Refusal-Calibrated Group Relative Policy Optimization (RC-GRPO), a reinforcement learning strategy that enables Multimodal Large Language Models to effectively reject nonexistent objects in Generalized Referring Expression Comprehension without compromising their localization accuracy on existing targets.

Original authors: Xuzheng Yang, Jun Ling, Tao Huang, Caiyan Qin, Peng Wang

Published 2026-08-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xuzheng Yang, Jun Ling, Tao Huang, Caiyan Qin, Peng Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to play a game of "I Spy" with a human. The human points to a picture and says, "Find the red ball." If the red ball is there, the robot should point right at it. But what if the human says, "Find the red ball," and the picture actually shows a blue cat? A smart robot should say, "I can't find a red ball here; there isn't one." However, many of today's super-smart computer brains (called Multimodal Large Language Models) are like eager students who are terrified of being wrong. They are so desperate to give an answer that when they don't see the red ball, they just guess anyway, pointing at the blue cat and insisting, "Here it is!" This is called a "hallucination"—making up facts that aren't true.

This paper tackles a specific, tricky version of this problem called "Generalized Referring Expression Comprehension." It's a fancy way of saying: "Can the robot find the object if it's there, and can it honestly say 'I don't see it' if it's not?" The researchers found that while these robots are great at finding things, they are terrible at admitting when things are missing. If you just train them to say "no" more often, they get too scared and stop finding the things that are actually there. The team needed a way to teach the robot to say "no" only when it's truly necessary, without making it forget how to say "yes" when it's right.

The authors of this paper propose a clever new training method called Refusal-Calibrated Group Relative Policy Optimization (RC-GRPO). Think of the robot's learning process like a game of "Hot and Cold." Usually, the robot gets a score based on how close its guess is. But when the object isn't there, the robot keeps guessing anyway, so it never learns the right answer. The researchers' first trick was to force the robot to say "None, I don't see it" during its practice runs, even if it wanted to guess. This gave the robot a clear "reward" for saying the right thing when the object was missing.

However, they realized that if they just praised the robot for saying "None," it would start saying "None" for everything, even when the object was there. To fix this, they added a special rule: if the robot says "None" when an object is actually present, it gets a big penalty. It's like telling the robot, "It's okay to say you don't see the ball if the ball isn't there, but if you say you don't see it when it's right in front of you, you lose points." They also added a second stage where the robot has to explain why it thinks the object is missing (e.g., "I don't see a red ball because there is only a blue cat"). This forces the robot to actually understand the picture rather than just memorizing a pattern.

The results show that this method works really well. When they tested their new method on three different challenging tests, the robots became much better at both finding objects and refusing to guess when they weren't there. For example, on one test, their method improved the robot's overall accuracy from 45% to 62%, and it even beat other advanced methods by a significant margin. The paper suggests that this approach helps the robot find a perfect balance: it stops making up fake locations for missing objects but doesn't lose its ability to find real ones. It's a step toward making AI that is not just smart, but also honest about what it sees and what it doesn't.

Technical Summary: Teaching MLLMs to Say No via Refusal-Calibrated GRPO

Problem Statement

The paper addresses the underexplored challenge of Generalized Referring Expression Comprehension (GREC). Unlike traditional Referring Expression Comprehension (REC), which assumes the target object always exists, GREC requires Multimodal Large Language Models (MLLMs) to perform two distinct tasks:

Positive Localization: Accurately localize an object described by a textual expression when it exists in the image.
Negative Refusal: Correctly identify and refuse to localize when the described object is absent or when there is a mismatch in attributes or relations (e.g., "white horse" in an image of a "brown horse").

Current MLLMs excel at localization but frequently fail at refusal. Due to the absence of negative samples during large-scale pre-training, these models tend to hallucinate bounding boxes for nonexistent objects rather than outputting a refusal (e.g., "None"). Existing post-training solutions, such as Supervised Fine-Tuning (SFT) and standard Reinforcement Learning (RL) methods like Direct Preference Optimization (DPO) or Group Relative Policy Optimization (GRPO), often improve refusal rates but at the cost of severely degrading localization accuracy on positive samples. This creates an optimization imbalance where models learn to indiscriminately say "no," rendering them unusable for practical applications requiring both capabilities.

Methodology: Refusal-Calibrated GRPO (RC-GRPO)

The authors propose RC-GRPO, a two-stage reinforcement learning framework designed to calibrate refusal behavior while preserving core localization competence. The framework builds upon GRPO but introduces specific mechanisms to address the unique challenges of GREC.

1. Training Data Construction
The method utilizes FineCops-Ref, a fine-grained benchmark containing hard negative samples generated via linguistic perturbations (changing attributes/relations) and image editing. The training set consists of 2,000 balanced samples (1,000 positive, 1,000 negative).

2. Stage I: Refusal Calibration with Contrastive Reward
This stage aims to establish a basic rejection capability (transitioning from 0 to 1) without collapsing positive performance. It employs three key components:

Forced-Rejection Rollout: Since base MLLMs rarely explore "None" outputs during early training, standard GRPO fails to compute valid advantages for negative samples (as all rollouts produce bounding boxes). RC-GRPO enforces "None" outputs via constrained beam search during rollouts if the model fails to generate them naturally. This ensures valid advantage estimation for negative samples.
Contrastive Reward: A reward function is designed to penalize over-refusal on positive samples.
- For positive samples: A correct bounding box (IoU $\ge$ 0.5) yields a reward of 1; a "None" prediction yields -1.
- For negative samples: A "None" prediction yields 1; a bounding box yields 0.
  This structure teaches the model that a poor box is preferable to a false refusal on positive cases.
Negative Advantage Scaling: To prevent the model from overfitting to the easier task of saying "None," the advantages of negative samples are scaled down by a coefficient $\alpha \in (0, 1]$ . This balances the optimization dynamics between the two sample types.

3. Stage II: Reason-Guided Reinforcement
To ensure refusals stem from genuine semantic understanding rather than superficial pattern matching, a second stage encourages the model to generate explanations for its refusals.

Rationale Reward: The model is prompted to explain why an object is absent. A rule-based verification system evaluates the generated explanation against the specific semantic perturbation (e.g., identifying that "behind" was changed to "in front of").
Reward Formulation: The reward includes a base reward for correct refusal plus a dense reward for correctly identifying the specific mismatch (Incorrect Pattern) and the original valid description (Original Pattern). This reinforces causal understanding and interpretability.

Key Contributions

Task Definition: The work formally studies the GREC task, highlighting the critical need to balance localization accuracy with refusal reliability for real-world deployment.
RC-GRPO Framework: The proposal of a calibrated RL framework that introduces Refusal Calibration (enforcing "None" outputs for valid gradient signals) and a Positive-Sample Penalty (contrastive reward) to jointly optimize rejection and localization.
Reasoning Reinforcement: The introduction of a reason-generation stage that grounds refusals in causal reasoning, improving both the reliability of the "No" decision and the interpretability of the model.
Empirical Validation: Extensive experiments demonstrating that RC-GRPO achieves a superior trade-off between positive and negative sample performance compared to SFT, DPO, and other RL baselines.

Experimental Results

Experiments were conducted on three benchmarks: FineCops-Ref, gRefCOCO, and D3 (Described Object Detection).

Performance Gains: On gRefCOCO, RC-GRPO applied to Qwen2.5-VL-7B improved Precision (Pr) from 38.2% (base) to 62.8% (RC-GRPO-I) and 64.9% (RC-GRPO-II), significantly outperforming specialist models and existing MLLM baselines like ROD-MLLM and CRS.
Balanced Trade-off: Unlike SFT, which achieved high refusal accuracy (N-acc) but caused a catastrophic drop in positive localization (P-acc), RC-GRPO maintained high P-acc while achieving robust N-acc.
Pareto Frontier: Analysis of the P-acc vs. N-acc trade-off showed that RC-GRPO establishes a superior Pareto frontier in the high-utility region, avoiding the "utility collapse zone" where models sacrifice grounding ability for refusal.
Generalization: The method showed negligible degradation on general multimodal benchmarks (MME, MMBench, POPE), indicating that the refusal logic does not compromise general visual understanding or induce new hallucinations.
Ablation Studies: Removing any component (Forced-Rejection, Contrastive Reward, or Advantage Scaling) resulted in suboptimal performance, confirming the necessity of the full framework.

Significance and Claims

The paper claims that RC-GRPO resolves the inherent optimization imbalance in training MLLMs for GREC. By internalizing causal refusal logic through calibrated reinforcement learning, the method enables models to "say no" reliably without sacrificing their core competence in localization. The authors assert that this approach moves beyond simple thresholding or architectural modifications, providing a scalable post-training solution that makes GREC substantially more practical for real-world scenarios where objects may be absent or mismatched. The work emphasizes that the model learns when and why to refuse, rather than indiscriminately rejecting inputs.