💻 computer science

OptiSight: Bridging Semantic Reasoning and Geometric Control for Embodied Navigation

OptiSight is a hybrid framework that integrates Vision-Language Model reasoning with deterministic visual servoing to enable efficient, zero-shot autonomous indoor navigation within an 8GB VRAM budget by leveraging Grounded-SAM for target localization and camera geometry for control without dense mapping.

Original authors: Alperen Avan, Jordi Sanchez-Riera

Published 2026-08-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Alperen Avan, Jordi Sanchez-Riera

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Moving through a room is something humans do without thinking, yet for a machine, it is a complex puzzle of two very different kinds. One part is seeing and understanding: knowing that a dark rectangle in a wall is a door, or that a pile of boxes blocks the path. The other part is moving: calculating exactly how far to turn the wheels to avoid a chair without bumping into it. For years, robots have been good at one or the other, but rarely both at the same time. Some systems are excellent at mapping a room and avoiding obstacles, but they cannot understand a command like "go find the red chair." Others can understand language and recognize objects, but they are often too slow or clumsy to move safely in real time. The challenge for researchers has been to build a robot that can reason about what it sees and then move with the precision of a human hand, all while running on a computer small enough to fit on a robot's back.

A team of researchers has developed a new system called OptiSight that bridges this gap. Instead of trying to make a single computer program do everything at once, they split the job into two distinct roles that work together. The first role is a "thinker," a powerful artificial intelligence that looks at the camera's view and figures out what objects are there and where the robot needs to go. The second role is a "pilot," a fast, simple set of rules that takes the thinker's decision and immediately steers the robot. The key innovation is that the thinker does not need to speak constantly. It only steps in when the robot reaches a major decision point, such as when it needs to find a new target or when it gets lost. Once the path is clear, the pilot takes over, guiding the robot smoothly to the destination without needing to ask for help again. This approach allows the robot to understand complex instructions like "get out of the room" while moving quickly and safely, all within the memory limits of a standard portable computer.

To test this idea, the researchers placed their system inside a detailed virtual world called AI Habitat, which simulates real indoor environments with furniture, walls, and obstacles. They gave the robot a simple but difficult task: to leave a room. They ran this test in twenty-four different scenarios, ranging from empty rooms to spaces filled with confusing objects, reflective surfaces, and narrow passages. In some tests, the robot had to navigate around a single obstacle; in others, it had to distinguish between two similar-looking doorways or find a path when the view was partially blocked. The system used a specific type of artificial intelligence to identify objects and a separate geometric system to calculate the exact angles needed to move. The robot was programmed to check its surroundings, decide where to go, and then execute the movement in a continuous loop.

The results showed that this split approach worked remarkably well. In twelve of the scenarios, the robot successfully completed the task in at least sixty percent of the attempts, and in several cases, it succeeded every single time. When the robot succeeded, it typically asked the "thinker" for help only once or twice during the entire journey. This proved that the system did not need to constantly re-evaluate the scene to move forward. Instead, it could rely on its geometric pilot to handle the continuous motion. The few times the robot failed were usually because it got too close to an obstacle and had to stop to recover, or because the environment was so confusing that the initial plan was invalid. Even in these difficult cases, the system showed it could recognize the problem and try again, rather than crashing blindly.

What makes this finding significant is how it balances intelligence with speed. The researchers found that by limiting the use of the heavy, slow-thinking artificial intelligence to only the most critical moments, they could keep the robot moving in real time. The system operated within a strict memory limit of eight gigabytes of video memory, a constraint that mimics the limited computing power found on many real-world robots. This means the approach is not just a theoretical idea for powerful supercomputers, but a practical method that could run on the small computers attached to actual machines. The experiments demonstrated that a robot does not need to be a genius at every single step to be effective; it just needs to know when to think and when to act. By separating the reasoning from the movement, OptiSight offers a reliable way for machines to navigate the messy, unpredictable world of a human home.

Technical Summary: OptiSight

Problem Statement

Autonomous indoor navigation in unstructured environments requires a dual capability: precise geometric control for collision-free movement and semantic understanding to interpret high-level instructions (e.g., "get out of the room"). Traditional navigation systems relying on Simultaneous Localization and Mapping (SLAM) and geometric path planning lack the ability to interpret open-vocabulary semantic concepts or natural language instructions. Conversely, while Vision-Language Models (VLMs) offer powerful zero-shot semantic reasoning and open-vocabulary perception, they are computationally expensive, introduce significant inference latency, and lack the geometric precision required for real-time, continuous motion control. Existing approaches often attempt to bridge this gap through end-to-end reinforcement learning (which suffers from sim-to-real transfer issues) or by continuously querying VLMs throughout navigation (which creates prohibitive computational overhead). The core challenge addressed by this work is how to integrate the semantic reasoning of VLMs with the efficiency and robustness of deterministic geometric control without incurring the latency or resource costs that prevent deployment on edge devices.

Methodology

The authors propose OptiSight, a hybrid autonomous navigation framework that decouples high-level semantic reasoning from low-level geometric control. The system operates within the AI Habitat simulation environment and is designed to execute the instruction "Get out of the room."

1. Finite-State Machine (FSM) Architecture

OptiSight employs a state-driven Chain-of-Thought (CoT) architecture comprising five distinct states: Search, Find, Scan, Navigate, and Recover.

Search: The VLM analyzes the current RGB image to determine if the semantic target (e.g., a doorway) is visible. If not, the robot performs an autonomous scanning procedure (nine 10° rotations) until the target is found.
Find: Upon detection, Grounded-SAM performs open-vocabulary segmentation to localize the target. The segmentation is refined over multiple inference passes to ensure stability. The 2D image coordinates are then projected into the 3D camera reference frame to estimate spatial location.
Scan: The camera tilts downward by 40° to maximize floor visibility. Grounded-SAM segments potential obstacles, which are projected into 3D space. If an obstacle is within a 0.50 m safety margin, a collision-free bypass trajectory is generated; otherwise, a direct trajectory is computed.
Navigate: The robot follows the planned trajectory using proportional visual servoing based on 3D waypoints. The system monitors the path continuously. Waypoints are discarded once the robot approaches within 0.30 m.
Recover: If the target is lost or the path becomes invalid, the system reverts to the Search state, re-invoking the VLM to re-establish the objective.

2. Selective VLM Integration

A critical design choice is the dual-isolated system architecture. The VLM (running in a separate environment) is queried only at specific state transitions (primarily Search and Recover) rather than continuously. This allows the geometric navigation pipeline (FSM, camera control, path planning) to operate deterministically and in real-time while semantic reasoning occurs in the background.

3. Geometric Control

The framework utilizes differentiable camera projection geometry to convert visual observations directly into navigation commands ( $\theta, \alpha$ ). This eliminates the need for dense semantic mapping or neural scene reconstruction, enabling precise closed-loop visual servoing with low computational cost.

Key Contributions

Hybrid Framework: OptiSight bridges semantic reasoning and geometric control by combining VLM-based open-vocabulary grounding with deterministic visual servoing.
State-Driven CoT: The paper introduces a finite-state machine that limits VLM inference to critical decision points, significantly reducing computational overhead compared to continuous reasoning approaches.
Efficient Semantic-to-Geometric Translation: The system leverages Grounded-SAM and camera projection to translate semantic targets into 3D navigation commands without requiring dense mapping or extensive training datasets.
Dual-Isolated Architecture: By separating the VLM and robotics pipelines via a socket-based interface, the system achieves compatibility between modern multimodal models and legacy robotics software, operating within an 8 GB VRAM budget.

Experimental Results

The framework was evaluated in AI Habitat across 24 diverse indoor navigation scenarios using two different VLMs: Qwen3.5-2B (Experiments 01–12) and Moondream2-2B (Experiments 13–24). The scenarios tested challenges including single/dual obstacle avoidance, semantic disambiguation, partial observability, extreme viewpoints, and reflective surfaces.

Success Rates: The framework achieved mission success (defined as completing the task in at least 50% of 10 runs) in 8 out of 12 scenarios with Qwen3.5-2B and 7 out of 12 with Moondream2-2B. Several scenarios (e.g., Exp 03, 07, 09, 19, 20, 23, 24) achieved 100% success rates.
Efficiency: Successful scenarios typically required only 1 to 3 VLM requests, demonstrating the effectiveness of the state-driven approach. In contrast, failed scenarios often involved higher VLM request counts (up to 8) and multiple recovery actions.
Robustness: The system successfully handled obstacle avoidance and semantic ambiguity. However, performance degraded in scenarios with extreme geometric distortions or very close obstacle proximity (e.g., minimum obstacle distance < 0.15 m), leading to collisions and mission failure.
Resource Usage: The system operated within the specified 8 GB VRAM limit, validating its suitability for resource-constrained edge platforms.

Significance and Claims

The paper claims that OptiSight provides a viable solution for zero-shot semantic navigation that does not rely on end-to-end policy learning or computationally expensive dense mapping. By separating high-level cognition from low-level control, the framework achieves a balance between the semantic flexibility of VLMs and the real-time performance of classical geometric navigation.

The authors emphasize that their approach enables explainable navigation through structured state transitions while maintaining the computational efficiency necessary for deployment on edge devices like the NVIDIA Jetson. The results suggest that for many embodied navigation tasks, continuous VLM inference is unnecessary; instead, strategic, state-triggered reasoning combined with deterministic geometric control offers a more practical and robust pathway for autonomous agents in complex indoor environments. Future work is noted to focus on extending the framework to broader tasks and physical robotic validation.