💻 computer science

EagleVision: A Dual-Stage Framework with BEV-grounding-based Chain-of-Thought for Spatial Intelligence

EagleVision is a dual-stage framework that enhances video-based spatial reasoning by combining a geometry-aware keyframe selection mechanism with an active, Bird's-Eye-View-grounded Chain-of-Thought process that iteratively retrieves new viewpoints to verify hypotheses, achieving state-of-the-art performance without requiring human-annotated reasoning traces.

Original authors: Jiaxu Wan, Xu Wang, Mengwei Xie, Hang Zhang, Mu Xu, Yang Han, Hong Zhang, Ding Yuan, Yifan Yang

Published 2026-03-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiaxu Wan, Xu Wang, Mengwei Xie, Hang Zhang, Mu Xu, Yang Han, Hong Zhang, Ding Yuan, Yifan Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a mystery in a giant, dark warehouse. You have a security camera feed, but it's a long video where the camera just spins around randomly. You need to answer a tricky question: "Which object is closer to the bed: the chair or the sofa?"

If you ask a standard AI (a "Multimodal Large Language Model") this question, it usually makes one of two mistakes:

The Guessing Game: It looks at a few random frames, guesses "Sofa," and moves on. It never actually checks if the chair is even in the room.
The Text-Only Thinker: It tries to reason in its head, saying, "Hmm, I can't find the chair in the pictures I have, but I'll guess the sofa anyway." It lacks the ability to go look for the missing evidence.

EagleVision is like a super-smart detective who refuses to guess until they have all the facts. It uses a two-step process to solve spatial puzzles in videos, and it does it by "thinking with space."

Here is how EagleVision works, explained with simple analogies:

Stage 1: The "Smart Snapshot" (Macro Perception)

Imagine you have a 10-minute video of a room, but you can only look at 5 snapshots because your brain (or computer memory) has a limit on how much it can process at once.

The Problem: If you just take 5 random snapshots, you might get 5 pictures of the same wall. You'd miss the bed, the chair, and the sofa entirely.
The EagleVision Solution: EagleVision uses a special algorithm called SPF-DPP. Think of this as a "Smart Camera Operator."
- It looks at the whole video and asks two questions for every frame:
  1. Is this picture relevant to the question? (Semantic Relevance: "Do I see a bed?")
  2. Is this picture from a different angle? (Geometric Diversity: "Have I already seen this angle?")
- It picks the perfect 5 snapshots that show you the most different parts of the room while still being relevant to your question. It ensures you have a "3D map" of the room in your mind before you even start thinking.

Stage 2: The "Active Detective" (Micro Verification)

Now that you have your 5 snapshots, you start thinking. But what if you still aren't sure? Maybe the chair is hidden behind a curtain in your snapshots.

The Problem: Normal AIs are stuck with the 5 snapshots they already have. They can't ask for more.
The EagleVision Solution: EagleVision has a Bird's-Eye View (BEV) Map. Imagine a flat, 2D map of the room floating above the video, showing exactly where the camera was for every frame.
- The "Hypothesize-Look-Verify" Loop:
  1. Think: The AI says, "I think the chair is near the window, but I can't see it in my current pictures."
  2. Ask: Instead of guessing, it points to the BEV Map and says, "I need to see the view from this specific spot on the map."
  3. Look: The system instantly jumps to the real video frame that matches that spot on the map and shows it to the AI.
  4. Verify: The AI looks at the new picture, updates its theory, and repeats the process until it is 100% sure.

How Does It Learn? (The "No-Teacher" Training)

Usually, to teach an AI to be a good detective, you need a human teacher to write out step-by-step instructions like: "First look at the bed, then look left, then look right." This is expensive and hard to do.

EagleVision learns differently. It uses Reinforcement Learning (like training a dog with treats).

The AI tries to solve the puzzle.
If it guesses the right answer, it gets a "treat" (a reward).
If it asks for a view that doesn't exist (like pointing the camera at a wall where no video was taken), it gets a "scolding" (a penalty).
Over time, the AI learns on its own: "Oh, I should only ask for views that actually exist in the video, and I should keep looking until I'm sure." It learns the strategy of searching without needing a human to write the script.

Why Is This a Big Deal?

It's Efficient: It doesn't waste energy looking at useless frames. It picks the best ones first.
It's Active: It doesn't just stare at what it's given; it actively seeks out the missing pieces of the puzzle.
It's Accurate: In tests, EagleVision beat almost every other open-source AI at understanding 3D space, distances, and layouts.

In a nutshell: EagleVision is the difference between a student who guesses the answer because they didn't study, and a student who opens the textbook, flips to the exact page they need, checks the diagram, and then confidently writes the correct answer. It turns "guessing" into "investigating."

1. Problem Statement

Video-based spatial reasoning (e.g., estimating distances, judging relative directions, or understanding room layouts) presents unique challenges for Multimodal Large Language Models (MLLMs):

Information Distribution: In long videos, critical geometric cues are unevenly distributed. Uniform frame sampling often misses key viewpoints necessary for 3D inference.
Static Context: Existing MLLMs are constrained by fixed token budgets and cannot request additional views once reasoning begins. If the initial evidence is insufficient, the model cannot "look again."
Lack of Active Reasoning: Current approaches either rely on static 3D features (point clouds) which lack active querying capabilities, or use 3D reconstruction as a black-box preprocessing step that cannot be iteratively refined during reasoning.
Data Scarcity: Collecting human-annotated multi-step spatial reasoning trajectories (Chain-of-Thought) is impractical, requiring a method that learns from answer-level supervision alone.

2. Methodology: EagleVision Framework

EagleVision introduces a dual-stage framework that combines geometry-aware frame selection with active, Bird's-Eye-View (BEV)-grounded reasoning. The system operates on a video and a textual query, utilizing a pre-computed BEV map and pose database derived from an off-the-shelf SLAM system.

Stage 1: Macro Perception (Keyframe Selection)

The goal is to select a compact set of keyframes that maximize both semantic relevance to the query and geometric diversity of viewpoints, under a strict token budget.

SPF-DPP (Semantics–Perspective-Fusion Determinantal Point Process):
- Geometric Component: Uses camera poses ($SE(3)$) to construct a sparse viewpoint graph. A heat-kernel diffusion is applied to the graph Laplacian to create a Diffusion Viewpoint Kernel ( $K_{view}$ ), which encodes global geometric similarity and viewpoint coverage.
- Semantic Component: Uses FG-CLIP to compute semantic scores between frames and the query text, normalized into a Quality Matrix ( $Q$ ).
- Selection: The final kernel $L_{dpp} = Q K_{view} Q$ is used to greedily select $k$ frames. This ensures the selected set is diverse in perspective (avoiding redundant views) while remaining highly relevant to the query.

Stage 2: Micro Verification (Active Spatial CoT)

The model performs iterative Spatial Multimodal Chain-of-Thought (Spatial MCoT) reasoning.

Closed-Loop Cycle: At each step, the model can:
1. Generate text (reasoning).
2. Active Query: Predict a 2D pose $(x, y, \theta)$ on the BEV plane to retrieve a new video frame.
3. Terminate with an answer.
BEV-Grounded Pose Querying: When the model predicts a pose, the system retrieves the real video frame closest to that pose using a scale-aware distance metric. If no frame covers the pose, an error prompt is returned.
Training (Zero-RL): The querying policy is trained using Group Relative Policy Optimization (GRPO).
- Reward Function: Combines standard task rewards (accuracy, format, tool usage) with a novel Spatial Grounding Reward.
- Spatial Reward: Penalizes trajectories where the model queries poses that fall outside the camera trajectory (i.e., regions with no coverage), forcing the model to learn valid querying strategies without human-annotated CoT traces.

3. Key Contributions

Dual-Stage Framework: Proposes EagleVision, the first framework to combine geometry-aware frame selection (Macro) with active, BEV-grounded iterative reasoning (Micro) for video spatial understanding.
SPF-DPP: Introduces a novel sampling strategy that jointly optimizes semantic relevance and viewpoint diversity using Determinantal Point Processes, solving the token budget vs. information density trade-off.
BEV-Grounded Active Reasoning: Formulates spatial reasoning as a tool-augmented process where the model actively queries specific viewpoints on a BEV map. This is trained purely via Reinforcement Learning with a spatial grounding reward, eliminating the need for expensive human reasoning annotations.
State-of-the-Art Performance: Demonstrates that EagleVision achieves SOTA results among open-source VLMs on both VSI-Bench and SQA3D, outperforming proprietary models and other specialized spatial reasoning methods.

4. Experimental Results

Benchmarks: Evaluated on VSI-Bench (3D spatial understanding) and SQA3D (situated QA).
Performance:
- On VSI-Bench, EagleVision achieved an average score of 63.5, ranking 1st among open-source VLMs. It outperformed the base model (Qwen3-VL-8B) by +4.1 points and the previous SOTA open-source model (VLM-3R-7B) by +2.6 points, despite not using a heavy 3D encoder.
- On SQA3D, it achieved 60.3% (EM@1), significantly outperforming methods like Struct2D (+1.8%) and SpatialMind (+14.0%).
Ablation Studies:
- Architecture: Removing any of the three core components (MCoT, BEV grounding, SPF-DPP) resulted in performance drops, confirming their complementary nature.
- SPF-DPP: Combining geometric diffusion and semantic modulation yielded the best results, proving the necessity of balancing diversity and relevance.
- Robustness: The system showed graceful degradation under pose noise (e.g., only a 0.4% drop with medium noise) and remained effective when the backend SLAM was replaced with a learning-based alternative (VGGT).

5. Significance

EagleVision represents a paradigm shift in video spatial reasoning by moving from passive consumption of fixed frames to active, hypothesis-driven exploration.

Efficiency: By decoupling heavy 3D reconstruction (done once offline) from the lightweight 2D VLM reasoning, it offers a cost-effective solution for multi-query scenarios.
Scalability: The ability to learn spatial querying purely from answer-level supervision via RL makes the approach scalable to large datasets where human reasoning traces are unavailable.
Generalization: The framework demonstrates strong cross-benchmark generalization, suggesting that the "hypothesize-look-verify" cycle is a fundamental capability for spatial intelligence that transfers across different tasks and datasets.

In summary, EagleVision effectively bridges the gap between 2D visual understanding and 3D spatial reasoning by enabling models to "think with space" through active view selection and geometric grounding.