Self-Questioning Vision-Language Models: Reinforcement Learning for Compositional Visual Reasoning

Technical Summary: Self-Questioning Vision-Language Models

Problem Statement
Vision-Language Models (VLMs) excel at direct pattern recognition tasks, such as object identification or scene description, but struggle with compositional visual reasoning. These are multi-step problems requiring the chaining of intermediate operations—for example, locating multiple objects, counting them, and comparing the results. Existing approaches to improve this reasoning rely heavily on human-annotated step-by-step explanations (e.g., Chain-of-Thought prompting or Supervised Fine-Tuning), which are expensive to create and difficult to scale. Furthermore, visual programming approaches depend on pre-defined tool libraries, limiting generalization to novel question types. The core challenge is enabling VLMs to discover and utilize intermediate reasoning steps without explicit human demonstrations.

Methodology
The authors propose a self-questioning framework trained via Group Relative Policy Optimization (GRPO), a reinforcement learning (RL) algorithm. The framework operates as follows:

• Model Architecture: The study utilizes Qwen2.5-VL-3B-Instruct, a 3-billion-parameter VLM. To make training feasible on limited hardware, the authors employ Low-Rank Adaptation (LoRA), updating less than 1% of the parameters while freezing the original weights.
• Training Objective: Unlike standard RL which rewards only the final answer, this framework enforces a structured output format. The model is prompted to generate a sequence of <sub-question> and <sub-answer> pairs before producing a <final-answer>.
• Reward Function: The reward signal is binary and strict. A model completion receives a reward of +1.0 only if it satisfies two conditions: (1) it strictly adheres to the sub-question/answer format (containing at least one pair), and (2) the final answer matches the ground truth. Otherwise, it receives -1.0.
• No Human Demonstrations: Crucially, the model is never shown examples of how to decompose questions. The behavior of breaking down complex queries into sub-questions must emerge solely from the reward signal incentivizing the format and correctness.
• Baseline Comparison: To isolate the effect of the self-questioning format from general RL benefits, the authors train a Direct+GRPO baseline. This model uses the same GRPO algorithm and datasets but is prompted to provide only a direct answer, with a reward function checking only for answer correctness and conciseness.

Key Contributions

1. Emergent Decomposition: The paper demonstrates that a VLM can learn to decompose compositional visual questions into sub-question–answer pairs entirely through reinforcement learning, without any human-authored reasoning traces.
2. First Application of GRPO to VLMs: This work is the first to apply Group Relative Policy Optimization (GRPO) to a vision-language model for compositional reasoning, adapting the method from text-only LLMs (like DeepSeek-R1) to the multimodal setting.
3. Rewarding Structure: The authors introduce a reward function that explicitly penalizes the lack of intermediate decomposition, making the structural breakdown of the problem a primary training objective alongside answer correctness.

Experimental Results
The framework was evaluated on two benchmarks: CLEVR (synthetic geometric scenes) and A-OKVQA (real-world photographs requiring external knowledge).

• RL Improves Accuracy: Both the Self-Questioning (SQ) and Direct RL models significantly outperformed the untrained base model on A-OKVQA. The base model achieved 46.8% accuracy, while Direct+GRPO reached 51.6% and SQ+GRPO reached 52.2%.
• Marginal Benefit of Self-Questioning: On A-OKVQA, the self-questioning format provided only a marginal gain of +0.6% over standard Direct RL. The authors conclude that the majority of the improvement (+4.8%) stems from the RL training signal itself (biasing the model toward correct answers) rather than the decomposition strategy.
• Cross-Domain Transfer: A significant finding is the transferability of the decomposition strategy. A model trained on synthetic CLEVR data using self-questioning improved by +2.6% when tested on real-world A-OKVQA images. In contrast, the Direct+GRPO model trained on CLEVR showed only a +0.2% gain on A-OKVQA. This suggests that the strategy of decomposing questions is a domain-general skill that transfers better than the specific answer-level reinforcement learned by standard RL.
• Format Tax on Simple Tasks: On simpler tasks (e.g., CLEVR), self-questioning introduced a "format tax." The SQ-trained model dropped to 81.0% accuracy on CLEVR when forced to use the sub-question prompt, compared to 97.4% for the base model. However, when evaluated with a direct prompt, the SQ model's accuracy recovered to 97.6%, indicating the model's visual capabilities remained intact but were hindered by the unnecessary overhead of decomposition on simple queries.

Significance and Claims
The paper claims that teaching AI systems to ask themselves intermediate questions is a promising strategy for complex visual reasoning, particularly when the difficulty of a question warrants explicit step-by-step decomposition.

• Scalability: The framework offers a scalable alternative to costly human annotations for multi-step reasoning, as the decomposition behavior emerges from reward signals alone.
• Conditional Value: The authors modestly note that the value of self-questioning is contingent on question complexity. It provides clear advantages for tasks requiring multi-step reasoning (e.g., counting, material identification) and facilitates cross-domain transfer, but it can degrade performance on simpler tasks where decomposition creates unnecessary noise.
• Future Direction: The results suggest that practical deployment, especially in safety-critical domains like autonomous driving or medical imaging, may benefit from adaptive prompting strategies that selectively apply self-questioning only when the estimated question complexity justifies the overhead.