💻 computer science

Beyond the Eye: Efficient Multimodal Reasoning via Self-Regulated Implicit Visual Tools

The paper introduces Beyond the Eye (BEE), a novel multimodal model that integrates implicit visual tools through a two-stage training process of structured supervised fine-tuning and self-regulated reward-driven alignment, enabling adaptive tool invocation that significantly reduces inference latency while maintaining state-of-the-art performance in fine-grained visual perception.

Original authors: Xiuwei Chen, Quanlin Chen, Wentao Hu, Zisheng Chen, Kun Xiang, Zehua Ma, Mingyang Zhang, Jianhua Han, Hanhui Li, Hang Xu, Xiaodan Liang

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiuwei Chen, Quanlin Chen, Wentao Hu, Zisheng Chen, Kun Xiang, Zehua Ma, Mingyang Zhang, Jianhua Han, Hanhui Li, Hang Xu, Xiaodan Liang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot friend who loves to solve puzzles. But this robot has a weird habit: no matter how easy the puzzle is, it always insists on calling a team of external experts, opening a toolbox, and re-scanning the picture over and over again just to find a missing piece. It's like trying to find your keys in the living room, but instead of just looking, you call a construction crew to tear down the walls, scan the floor with lasers, and then rebuild the room before you can grab the keys. It works, but it's slow, expensive, and totally exhausting.

This is exactly the problem researchers found with current "Thinking with Images" AI models. They rely too much on calling external visual tools, which creates a massive delay and wastes computing power.

Enter BEE (Beyond the Eye), a new kind of robot brain that learned to stop over-calling the experts. Instead of running to the toolbox every time, BEE learned to look inside its own head first. It developed a "self-regulated" sense of its own limits. If the puzzle is easy, BEE says, "I got this," and solves it instantly using its own knowledge. If the puzzle is truly hard, then it quietly activates an "implicit tool"—a mental shortcut that simulates zooming in or rotating the image without actually stopping to call an outside program.

The Big Discovery
The main finding is that BEE can balance its own brainpower with these mental shortcuts so perfectly that it solves complex visual puzzles faster and more accurately than models that are much bigger or use real external tools. In fact, when tested on high-resolution images (8192 pixels), BEE was about 86.2% faster than the previous top method called DeepEyes. It's like switching from a snail that carries a heavy backpack to a cheetah that knows exactly when to sprint.

What BEE is NOT
The paper is very clear about what this isn't. It's not a model that just calls tools randomly or constantly. The researchers explicitly argue against the idea that models should always rely on external tools or that they should just "think harder" without knowing when to stop. They showed that older models often call tools even when they already know the answer, which is a waste of time. BEE is specifically designed to avoid this redundancy. It's not just a faster tool; it's a smarter decision-maker that knows when not to use a tool.

How They Taught It
The researchers didn't just tell BEE to be faster; they taught it in two fun stages:

The "Practice" Stage: They showed BEE thousands of examples where it had to decide whether to use a tool or not. They gave it a special "formalized" training guide (like a cheat sheet with slots for tools) to help it understand the difference between a problem it can solve alone and one that needs help.
The "Reward" Stage: This is the magic sauce. They introduced a metric called Net Tool Gain (NTG) to measure how much a tool actually helped. They found that the first version of BEE still called tools too much (like a kid who keeps asking for help even when they know the answer). So, they created a special reward system (MC-Reward) that gave BEE a "gold star" for solving hard problems with tools but a "time-out" for using tools on easy problems. This taught BEE to trust its own brain first.

The Results
The numbers speak for themselves. On a test called HRBench (which uses high-resolution images), BEE-7B (a model with 7 billion parameters) beat the previous open-source champions. Even more impressively, on a test called MME-RealWorld, BEE-8B and BEE-4B actually scored higher than the massive commercial model Kimi-K2.6, which has way more parameters.

But the real win is speed. While other models took seconds to process a single image because they were constantly calling external tools, BEE did it in a single pass. In tests, BEE-7B processed 1.61 samples per second, compared to just 0.07 samples per second for the DeepEyes model. That's a huge jump in efficiency.

What the Paper Doesn't Claim
The authors are careful not to say this is a magic bullet for every problem. They admit that for very simple tasks (like reading text or spotting basic shapes), BEE doesn't need to use its tools at all, and forcing it to do so would actually hurt performance. They also note that while BEE is great at "fine-grained" details (like spotting a tiny object in a crowded scene), it doesn't necessarily make the model better at things it was already good at, like basic logic or OCR, unless the task specifically requires visual manipulation.

In short, BEE is a robot that learned the art of "knowing what it doesn't know." It stopped wasting time calling for help on easy tasks and started using its mental tools only when the puzzle was truly tough. The result? A smarter, faster, and much more efficient way for AI to see the world.

Technical Summary: Beyond the Eye (BEE)

1. Problem Statement

Recent Multimodal Large Language Models (MLLMs) have advanced fine-grained perception through the "Thinking with Images" (TwI) paradigm, where models iteratively invoke external visual tools (e.g., zooming, cropping) to re-encode image regions. While effective, this approach suffers from two critical limitations:

Computational Overhead: Frequent external tool calls and repeated image re-encoding introduce substantial inference latency and I/O bottlenecks.
Lack of Capability Boundary Awareness: Existing models, including both explicit tool-based and emerging latent reasoning approaches, struggle to adaptively determine when to rely on internal parametric knowledge versus when to invoke tools. This leads to redundant tool usage on tasks the model can already solve, wasting computational resources without improving accuracy.

2. Methodology: Beyond the Eye (BEE)

The authors propose BEE, a novel Implicit Visual Tool (IvT) paradigm centered on self-regulated capability. Instead of calling external APIs, BEE folds visual tool invocation directly into the model's training objective, learning to generate implicit visual representations (semantic tokens) that simulate tool effects within a single forward pass.

The framework employs a two-stage training process:

Stage 1: Formalized Chain-of-Thought (CoT) Supervised Fine-Tuning (SFT)

Objective: Activate the model's implicit tool representations, adaptive switching mechanisms, and basic reasoning abilities.
Data Construction: The authors construct the BEE-SFT-320K dataset by integrating heterogeneous data from Refocus, Thyme, mini-o3, and Visual-CoT.
- Trajectory Polishing: Explicit tool invocations (code-based) are transformed into implicit representations. A pretrained semantic tokenizer encodes auxiliary images (e.g., zoomed/cropped views) into semantic tokens (<tool_pad>), replacing the code segments in reasoning trajectories.
- Mixed Invocation States: The dataset includes a mixture of reasoning trajectories with and without implicit tool slots. This explicitly guides the model to learn when to invoke tools and when to rely on internal knowledge.
- Auxiliary Alignment: An auxiliary caption prediction task is used to strengthen the model's ability to understand these new implicit visual tokens.

Stage 2: Self-Regulated Reward-Driven Alignment

Problem Identification: Analysis of the SFT-trained model reveals a lack of capability boundary awareness. The model tends to overuse tools even on tasks it can solve internally.
Net Tool Gain (NTG) Metric: To quantify this, the authors introduce the NTG metric, which measures the marginal contribution of tool invocation to task success ( $NTG = \frac{|G| - |L|}{|C_{tool}|}$ ), where $G$ is effective gain, $L$ is tool-induced hallucination, and $C_{tool}$ is the set of tool-invoked samples. The SFT model showed a low NTG (6.2%), indicating significant redundancy.
MC-Reward (Self-Regulated Confidence Calibration): A composite reward function is designed to penalize ineffective tool dependency and encourage knowledge routing.
- Dynamic Capability Signal: The model estimates task difficulty ( $p$ ) based on the average accuracy of sampled trajectories within a rollout group.
- Reward Logic:
  - Easy Tasks ( $p \geq \tau$ ): If the model solves the task without tools, it receives a high reward. If it unnecessarily invokes a tool, it is penalized.
  - Hard Tasks ( $p < \tau$ ): If the model invokes tools to solve a difficult task, it is rewarded. If it fails to use tools when needed, it is penalized.
- Implementation: The reward is formulated using smooth, continuous functions (including cosine gating) to ensure training stability, optimizing the policy via the DAPO algorithm.

3. Key Contributions

BEE Paradigm: A self-regulation-driven implicit visual tool framework that avoids redundant tool usage and reduces inference latency by internalizing tool behaviors.
Formalized CoT Labels & NTG Metric: Construction of a specialized SFT dataset with mixed invocation states to activate adaptive switching, alongside the NTG metric to quantitatively evaluate the marginal value of tool usage.
Self-Regulated Reward Mechanism: A novel reward objective (MC-Reward) that enables the model to dynamically assess its own cognitive boundaries, suppressing redundant tool calls on easy tasks while leveraging tools for difficult ones.
Empirical Validation: Demonstration that BEE achieves state-of-the-art performance in fine-grained visual perception while significantly improving inference efficiency compared to explicit tool-based and latent reasoning baselines.

4. Experimental Results

The authors evaluated BEE on multiple benchmarks, including V*, HRBench (4K/8K), MME-RealWorld, LogicVista, and POPE.

Performance:
- BEE-8B and BEE-4B surpassed the commercial model Kimi-K2.6 in average accuracy on perception tasks.
- BEE-8B achieved an average accuracy of 75.3 on real-world understanding benchmarks, outperforming the strongest open-source baseline (Qwen3-VL-235B) by 1.2 points.
- BEE consistently outperformed agentic models (e.g., DeepEyes, Thyme) and latent reasoning models (e.g., Monet, EVA) across various scales.
Efficiency:
- Latency Reduction: Under the 8K resolution setting, BEE achieved up to ~86.2% inference acceleration compared to the DeepEyes method.
- Throughput: BEE-7B achieved a throughput of 1.61 samples/s, compared to 0.07 samples/s for DeepEyes and 0.33 samples/s for DeepEyesV2.
- Cost: Operational costs were reduced from ~$6.40 (DeepEyes) to $0.72 per 1K samples.
Ablation Studies:
- Removing implicit tools or replacing them with meaningless tokens caused consistent performance drops, validating the necessity of the implicit tool design.
- The MC-Reward mechanism significantly improved the Net Tool Gain (NTG) from 6.2% (BEE-Static) to 66.0% (BEE-7B), confirming the effectiveness of the self-regulated mechanism in reducing redundancy.

5. Significance and Claims

The paper claims that BEE provides a promising direction for developing implicit tool MLLMs that combine strong cognitive ability with extremely low latency. By establishing a self-regulated capability boundary, BEE enables models to adaptively balance internal knowledge and implicit tool use, fundamentally addressing the trade-off between reasoning depth and inference efficiency. The authors assert that this approach advances the development of self-regulated MLLMs, offering a scalable alternative to the heavy computational overhead of traditional "Thinking with Images" paradigms.