🤖 AI

Discovering Failure Modes in Vision-Language Models using RL

This paper proposes a reinforcement learning-based framework that automatically trains an adaptive questioner agent to systematically discover and characterize novel failure modes in vision-language models by generating increasingly complex queries without human intervention.

Original authors: Kanishk Jain, Qian Yang, Shravan Nayak, Parisa Kordjamshidi, Nishanth Anand, Aishwarya Agrawal

Published 2026-04-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kanishk Jain, Qian Yang, Shravan Nayak, Parisa Kordjamshidi, Nishanth Anand, Aishwarya Agrawal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot assistant that can see pictures and answer questions about them. It's great at describing a sunset or identifying a dog. But, like any smart person, it has blind spots. Maybe it can't count how many cows are in a field, or it gets confused about which way a person is facing.

The problem is, how do we find these blind spots?

The Old Way: The Manual Search

Previously, researchers tried to find these weaknesses the old-fashioned way: by hand. They would sit down, think, "Hmm, what tricky questions can I ask this robot?" and then write a list of 100 questions to test it.

This is like trying to find a specific needle in a haystack by looking at one needle at a time. It's slow, expensive, and the humans doing the searching might miss things because they are biased. They might only look for the "big" mistakes and miss the tiny, subtle ones.

The New Way: The "Robot Detective" (This Paper)

This paper introduces a new, automated system using Reinforcement Learning (RL). Think of it as creating a Robot Detective whose only job is to trick the Target Robot into making mistakes.

Here is how the system works, using a simple analogy:

1. The Three Characters

The Target Robot (The Answerer): This is the AI we want to test. It tries to answer questions about an image.
The Detective (The Questioner): This is a special AI trained to ask questions. Its goal isn't to be helpful; its goal is to find the Target Robot's weaknesses.
The Referee (The Verifier): This is a super-smart AI that watches the game. It checks: "Did the question make sense?" and "Did the Target Robot get it wrong?"

2. The Game Loop (Training)

Imagine a video game where the Detective gets points (rewards) only when the Target Robot fails.

Round 1: The Detective asks, "What color is the car?" The Target Robot answers correctly. The Detective gets zero points.
Round 2: The Detective tries, "How many wheels are on the bike?" The Target Robot guesses "3." The Referee says, "Wrong! That's a failure!" The Detective gets points.
The Magic: Because the Detective gets points for finding mistakes, it learns to ask better and trickier questions over time. It stops asking easy questions and starts asking complex ones like, "What is the reflection of the red balloon in the window of the car parked behind the tree?"

3. The "Curriculum" of Failure

The paper found that this Detective doesn't just ask random questions. It learns a curriculum:

Early in training: It finds the "low-hanging fruit"—easy mistakes like counting errors or simple object recognition.
Later in training: Once it masters the easy stuff, it starts digging deeper. It begins asking questions that require spatial reasoning (e.g., "Is the cat to the left of the dog, or just behind it?") or complex logic (e.g., "If the light is red, why is the car moving?").

Why is this better?

The researchers compared their "Robot Detective" to other methods:

Static Benchmarks: Like a fixed test paper. Everyone takes the same test, so everyone learns the same answers.
The RL Detective: It's like a sparring partner. If you get good at blocking left, it learns to punch right. It adapts to the specific weaknesses of the robot it is fighting.

The Results: Finding 36 New "Blind Spots"

By using this automated detective, the researchers discovered 36 completely new types of failures that humans hadn't even thought to look for before.

They found that robots struggle with things like "Exclusion Reasoning" (understanding what is not in the picture) and "Anatomical Recognition" (knowing which way a body part is facing).
The questions generated by the RL system were much more diverse. Instead of asking "What is this?" 1,000 times, the system learned to ask "Where," "How many," "Why," and "What if," covering a much wider range of human-like thinking.

The Big Picture

This paper is a shift from static testing (giving a robot a fixed exam) to dynamic hunting (training a robot to find the exam questions that will break the other robot).

It's like upgrading from a teacher who grades a test with a red pen, to a coach who trains a sparring partner specifically to find your weak spots so you can get stronger. This helps us build Vision-Language Models that are not just "smart" on paper, but actually robust and reliable in the real world.

1. Problem Statement

Vision-Language Models (VLMs) have achieved strong performance on multimodal benchmarks but frequently fail on tasks humans find trivial, such as counting, spatial reasoning, and viewpoint understanding.

Limitations of Current Methods: Existing approaches to identifying these failure modes rely on manual benchmark design (e.g., MMBench, BLINK) or static "open-loop" automated probing using Large Language Models (LLMs).
- Manual methods are costly, unscalable, and subject to human cognitive biases (focusing on salient objects rather than subtle details).
- Static automated methods generate diverse queries but do not adapt to the specific responses of the target VLM, failing to capture nuanced, model-specific vulnerabilities.
Goal: The authors propose a fully automated framework to discover failure modes (blind spots) for any candidate VLM on a given data distribution without human intervention, specifically targeting fine-grained visual details and complex skill compositions.

2. Methodology

The authors formulate failure-mode discovery as a Reinforcement Learning (RL) problem. The system consists of three interacting components:

A. System Architecture

Questioner (RL Agent): A VLM-based agent trained to generate a set of $k$ questions ( $Q$ ) for a given image ( $I$ ). It uses Group Relative Policy Optimization (GRPO) to optimize its policy. The agent learns to adaptively probe the target model, increasing question complexity as training progresses.
Answerer (Target VLM): The candidate VLM whose failure modes are being discovered. Its parameters are frozen during training. It generates answers ( $A$ ) to the questions.
Verifier (Reward Model): A high-capacity reasoning VLM (e.g., Qwen3-VL-30B) that evaluates the interaction between the Questioner and Answerer to compute a reward signal.

B. Reward Function Design

The core innovation is a multi-component reward function ( $R_Q$ ) designed to encourage the generation of valid, diverse, and failure-inducing questions:
$R_Q = \Delta_Q \cdot V_Q \cdot (1 - C_Q) - P_Q$

Validity ( $V_Q$ ): Ensures questions are grammatically correct, atomic, visually grounded, and objective.
Correctness ( $C_Q$ ): The target VLM's answer is evaluated against the ground truth (or verifier's reasoning). The agent is rewarded only if the answer is incorrect ( $C_Q=0$ ).
Diversity ( $\Delta_Q$ ): Prevents the agent from "reward hacking" by repeating the same question types. It includes:
- Semantic Diversity: Penalizes questions semantically similar to previously found failures (using cosine similarity of embeddings).
- Lexical Diversity: Penalizes repetitive question prefixes (e.g., always starting with "How many") using inverse frequency counts.
Penalty ( $P_Q$ ): A negative reinforcement term for invalid questions to accelerate training.

C. Failure Taxonomy Pipeline

To analyze the discovered failures, the authors propose a four-stage automated pipeline:

Primitive Identification: LLMs identify the specific skills required to answer each question.
Topic Modelling: Clustering primitives into meaningful categories (using BERTopic).
Skill Extraction: Standardizing topic names across methods.
Meta-Skill Identification: Grouping skills into high-level "meta-skills" (e.g., Spatial Reasoning, Object Counting) to create a hierarchical taxonomy.

3. Key Contributions

Novel RL Framework: The first fully automated, adaptive framework to discover VLM failure modes. Unlike static benchmarks, the questioner learns to target specific weaknesses of the target model.
Discovery of Novel Failure Modes: The framework identified 36 novel failure modes (skills) that previous methods missed, including complex tasks like "Prominence Detection," "Threat Assessment," and "Path Tracking."
Compositional Complexity: The RL agent learns to generate questions requiring multi-hop reasoning and the combination of multiple skills (e.g., spatial localization + object counting + attribute reasoning), stress-testing VLMs beyond simple perception.
Comprehensive Evaluation: Extensive comparisons against baselines (Zero-Shot, ConMe, SFT, Expert Iteration) across metrics like Failure Discovery Rate (FDR), diversity, and skill coverage.

4. Experimental Results

The framework was evaluated on 1,000 images from the COCO dataset using various Qwen-VL models as questioners, answerers, and verifiers.

Failure Discovery Rate (FDR): The RL approach achieved a significantly higher FDR (47.58%) compared to the next best baseline, Expert Iteration (38.36%). The RL+SFT variant (fine-tuning before RL) reached 50.73%.
Diversity:
- Semantic Diversity: RL methods scored highest (64.96 vs. 61.78 for ConMe).
- Lexical Diversity: RL methods showed a massive improvement (0.53 vs. 0.19 for Expert Iteration), indicating they do not rely on repetitive templates.
Skill Coverage: RL discovered 131 unique skills, compared to 89–94 for baselines. It also probed more complex questions, with a higher percentage of questions requiring 4+ simultaneous skills.
Generalizability: The framework performed robustly across different model scales (3B to 8B parameters). While FDR decreased as the target model became stronger, RL consistently outperformed baselines and uncovered both universal weaknesses (e.g., Exclusion Reasoning) and model-specific blind spots (e.g., Negation Processing in smaller models).
Training Dynamics: Analysis showed an "implicit curriculum" where the agent first exploits easy failures (e.g., Numerical Reasoning) and progressively uncovers harder, late-emerging failures (e.g., Cultural/Symbolic Recognition) as training matures.

5. Significance and Conclusion

This work represents a paradigm shift from static, manually curated benchmarks to adaptive, RL-driven failure discovery.

Scalability: It eliminates the need for labor-intensive human annotation, allowing for continuous evaluation of VLMs as they evolve.
Depth of Insight: By focusing on fine-grained details and compositional reasoning, it reveals "cognitive blind spots" that standard benchmarks miss.
Future Impact: The authors suggest this framework can be extended to multi-turn dialogues and video understanding, and the discovered failure modes could be integrated into fine-tuning pipelines to actively improve model robustness.

In summary, the paper demonstrates that an RL-based agent can autonomously learn to "red-team" VLMs, uncovering a broader, deeper, and more diverse set of vulnerabilities than human-designed or static automated methods.