💻 computer science

See Further, Think Deeper: Advancing VLM's Reasoning Ability with Low-level Visual Cues and Reflection

This paper introduces ForeSight, a unified multimodal framework that enhances Vision-Language Model reasoning by integrating low-level visual tools and a mask-based visual feedback mechanism within a Reinforcement Learning paradigm, achieving state-of-the-art performance on the newly constructed CG-SalBench dataset.

Original authors: Zhiheng Wu, Tong Wang, Shuning Wang, Naiming Liu, Yumeng Zhang

Published 2026-04-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhiheng Wu, Tong Wang, Shuning Wang, Naiming Liu, Yumeng Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a tricky puzzle where you have to find the one object in a picture that is different from all the others. Most current AI models are like a person who tries to solve this puzzle by just talking about the picture. They describe what they see in words, but they often miss tiny details because they aren't actually "looking" closely enough. They might guess the answer and move on, never checking if they were right.

The paper introduces a new AI framework called ForeSight (which stands for "See Further, Think Deeper"). It teaches the AI to stop just talking and start actually doing things to understand the image better. Here is how it works, using simple analogies:

1. The Problem: The "Blind Talker"

Current AI models are like a detective who solves crimes by reading a report but never visits the crime scene. They rely on their internal memory (text) to guess what an image looks like. If the report is vague, they guess wrong. They also don't have a way to say, "Wait, I might be wrong," and go back to check.

2. The Solution: Giving the AI a "Toolbox" (See Further)

ForeSight gives the AI a set of special glasses and tools to examine the image up close, just like a human would. Instead of just guessing, the AI can decide to:

Zoom In: Like leaning in closer to read a tiny label on a jar.
Trace Edges (Canny Tool): Like using a highlighter to trace the outline of a shape to see exactly where it ends and begins.
Change Colors: Like putting on special sunglasses that make red things look bright blue, helping to spot a red apple in a pile of green ones.

The AI learns to ask itself: "Do I have enough information? No? Okay, let me use the Zoom tool." It only uses these tools when it thinks it needs them, making the process efficient.

3. The Secret Sauce: The "Self-Correction Mirror" (Think Deeper)

This is the most unique part. Most AIs give an answer and stop. ForeSight is different; it has a mirror.

The Process: The AI makes a first guess (a "draft answer").
The Mask: It then takes a digital "mask" (like a piece of black tape) and covers the part of the image it thinks is the answer.
The Check: It looks at the rest of the uncovered image. It asks: "Does the stuff I didn't cover look like the thing I'm looking for?"
- If the uncovered parts look exactly the same as the thing it picked, it says, "Okay, I'm right!"
- If the uncovered parts look different (e.g., it picked a red circle, but the uncovered area has a blue square), it says, "Wait, I made a mistake!" and changes its answer.

This turns the AI's thinking process from a one-way street (Question → Answer) into a loop (Question → Answer → Check → Fix Answer).

4. How It Learned: The "Practice Coach"

The researchers didn't just tell the AI to do this; they trained it using a method called Reinforcement Learning.

Think of this like a video game coach. The AI tries to solve the puzzle.
If it uses the right tools and gets the right answer, it gets a "point" (reward).
If it guesses without looking or gets the answer wrong, it gets no points.
Over thousands of tries, the AI learns the best strategy: "I should use the edge tool here, and I should always check my work with the mirror before I finish."

5. The Results: A Smarter, Smaller AI

The researchers tested this new AI on a new set of puzzles they created (called CG-SalBench).

The Surprise: They built this system on a relatively small AI model (7 billion parameters).
The Win: Even though it was smaller than some massive, famous AI models, ForeSight beat them all at finding the "odd one out" in pictures. It was even better at pinpointing the exact location of the object than models that are 10 times bigger.

Summary

In short, ForeSight is an AI that doesn't just guess based on words. It learns to grab a magnifying glass when it's confused and look in a mirror to check its own work. By doing these simple, human-like actions, it becomes much smarter at understanding pictures, proving that you don't need a giant brain if you have the right tools and the habit of double-checking your work.

1. Problem Statement

While Vision-Language Models (VLMs) have improved significantly with Reinforcement Learning (RL), existing methods face two critical limitations in complex visual reasoning tasks:

Neglect of Low-Level Visual Information: Current models often rely on high-level semantic understanding, missing fine-grained visual features (e.g., edges, subtle color shifts, texture details) crucial for tasks like anomaly detection. Text-only reasoning leads to errors when visual details are ambiguous.
Lack of Effective Visual Feedback (Open-Loop Reasoning): Most VLMs operate in an open-loop manner where the reasoning process is linear ("Reasoning $\to$ Answer"). Once an answer is generated, the model does not re-examine it against visual evidence. There is a lack of a closed-loop mechanism where the model can verify its own predictions using visual feedback to correct errors.

2. Methodology: The ForeSight Framework

The authors propose ForeSight, a unified multimodal interleaved reasoning framework driven by RL. It enables VLMs to "See Further" (via low-level tools) and "Think Deeper" (via visual reflection).

A. Core Components

Low-Level Visual Tools:
The framework integrates a toolkit of fundamental image processing tools that the model can autonomously invoke during the Chain-of-Thought (CoT) process:
- Canny Tool: Detects edges to clarify object boundaries and structural contours.
- Zoom-In (ROI) Tool: Allows the model to focus on specific regions to extract fine-grained details when global context is insufficient.
- Color Tool: Enhances color contrast to distinguish objects based on hue or saturation.
- Mechanism: The model dynamically decides when and which tool to use based on the current reasoning state, rather than using a fixed pipeline.
Mask-Based Visual Reflection:
To transform reasoning from open-loop to closed-loop, ForeSight introduces a reflection mechanism:
- After generating a draft answer (including a bounding box and character description), the model creates a mask covering the predicted region on the original image.
- The masked image (showing only the unmasked parts) is fed back into the model.
- The model analyzes the unmasked area to verify if the predicted object is truly unique compared to the rest. If inconsistencies are found, the model iteratively refines its answer.
- This creates a cycle: Reasoning $\to$ Draft Answer $\to$ Visual Feedback (Mask) $\to$ Reflection $\to$ Final Answer.

B. Training Strategy

The framework employs a two-stage training pipeline based on Group Relative Policy Optimization (GRPO):

Stage 1: Supervised Fine-Tuning (SFT) & Cold Start:
- A new dataset, CG-SalBench, is constructed based on the SalBench dataset. It contains "Odd-One-Out" tasks requiring precise bounding box localization and character identification.
- Data is partitioned into "tool-free" and "tool-dependent" subsets based on model performance.
- A larger model (Qwen2.5-VL-72B) generates high-quality CoT annotations with tool usage and reflection traces, which are cleaned to form a cold-start dataset for the 7B model.
Stage 2: Reinforcement Learning (RL):
- The model is optimized using a composite reward function that includes:
  - Format Consistency: Ensuring structured output.
  - IoU Reward: Maximizing bounding box overlap with ground truth.
  - F1 Reward: Improving character identification accuracy.
  - Tool Reward: Encouraging correct tool invocation and penalizing overuse or invalid formats.
  - Verification Reward: Aligning the reflection logic (e.g., if the draft is wrong, the reflection must identify the error).

3. Key Contributions

Closed-Loop Visual Reasoning: First to introduce visual feedback-based reflection into VLM reasoning, shifting from open-loop to a self-correcting closed-loop process.
Low-Level Visual Integration: Proposes an RL-driven mechanism for the autonomous invocation of low-level visual tools (Canny, Zoom, Color), enabling models to actively "see" details they previously ignored.
New Benchmark (CG-SalBench): Introduces a specialized dataset for evaluating "Odd-One-Out" grounding and character identification, bridging the gap between synthetic and real-world visual saliency tasks.
Efficiency over Scale: Demonstrates that a 7B parameter model (ForeSight-7B) can outperform much larger models (e.g., 72B) and closed-source SOTA models through superior reasoning architecture rather than just parameter scaling.

4. Experimental Results

Experiments were conducted on the CG-SalBench benchmark and general VQA tasks (MME, MMBench).

Performance on CG-SalBench:
- ForeSight-7B achieved an IoU of 62.24% and an F1 score of 84.24%.
- It significantly outperformed the base Qwen2.5-VL-7B (IoU +29.68%, F1 +19.27%).
- It surpassed the Qwen2.5-VL-72B (66.00% IoU) in F1 score and outperformed the best closed-source model (Doubao-Seed-1.6) in IoU by over 14%.
- It notably outperformed GPT-5, which had a high F1 but very low IoU (7.85%), indicating ForeSight's superior localization capabilities.
Ablation Studies:
- Tools + Reflection: Using both components together yielded the highest performance, confirming their complementary nature (Tools for perception, Reflection for verification).
- Training Stages: The combination of SFT (cold start) and RL was essential; RL alone was unstable, and SFT alone lacked the refinement of dynamic tool usage.
Generalization: ForeSight showed significant improvements on grounding benchmarks (RefCOCO series) and the MME VQA benchmark, validating its robustness across different visual tasks.

5. Significance

This paper challenges the prevailing "scaling law" paradigm in VLMs, suggesting that architectural innovation in reasoning processes (specifically integrating low-level cues and self-verification) is more effective than simply increasing model size. By enabling models to "See Further" (active perception) and "Think Deeper" (iterative verification), ForeSight provides a blueprint for building more robust, human-aligned, and reliable multimodal agents capable of handling complex, fine-grained visual reasoning tasks.