💻 computer science

LocalNav: Distilling Frontier VLMs and Embodied RL for On-Device Object Goal Navigation

This paper presents "LocalNav," a framework that distills complex spatial-semantic reasoning from frontier Vision Language Models into a lightweight 4B-parameter model and optimizes it with E-RLVR and quantization, enabling high-performance, low-latency object goal navigation on resource-constrained edge devices without relying on cloud execution.

Original authors: Nicolas Baumann, Liam Boyle, Pu Deng, Edoardo Ghignone, Boyang Sun, Marc Pollefeys, Luca Benini, Michele Magno

Published 2026-06-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nicolas Baumann, Liam Boyle, Pu Deng, Edoardo Ghignone, Boyang Sun, Marc Pollefeys, Luca Benini, Michele Magno

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, but very heavy, robot brain. This brain is like a giant library of knowledge that can understand complex instructions like, "Find the drawers under the microwave." The problem is, this brain is so big and heavy that it can't fit inside a small robot; it has to live in a giant cloud computer far away. Every time the robot needs to make a decision, it has to send a message to the cloud, wait for a reply, and then move. This is slow, expensive, and doesn't work if the robot is in a place with no internet.

The paper introduces LocalNav, a new way to shrink that giant brain down so it can fit inside a small robot (like one with a Jetson Orin chip) and work entirely on its own, without needing the cloud.

Here is how they did it, using three simple steps:

1. The "Shadow Student" (Distillation)

Think of the giant cloud brain (like Claude or GPT) as a Master Chef. The Master Chef can cook a perfect, complex meal, but it takes a long time and uses a huge kitchen. The researchers wanted to teach a Student Chef (a much smaller, 4-billion-parameter model called Qwen3.5-4B) how to cook the same dishes.

Instead of forcing the student to read millions of cookbooks, they just showed the student about 500 examples of the Master Chef solving navigation puzzles. They said, "Watch how the Master Chef thinks: 'I see a microwave, so the drawers must be below it.'" By copying these specific examples, the Student Chef learned to navigate just as well as the Master Chef, but with a brain small enough to fit in a backpack.

The Result: The small robot brain achieved a success rate of 34.5%, which is very close to the giant cloud brain's 39.7%.

2. The "Brevity Coach" (E-RLVR)

There was one catch: The Student Chef was a bit too chatty. When asked to find the drawers, the Master Chef might say, "I see a silver appliance that looks like a microwave, and I recall that drawers are usually below microwaves, so I will go there." That's a lot of words to type! For a robot with limited battery and processing power, typing all those words takes too long.

To fix this, the researchers used a technique called E-RLVR (Embodied Reinforcement Learning from Verifiable Rewards). Imagine a strict coach who watches the student robot.

If the robot takes a long time to answer or says too many words, the coach gives a "bad grade."
If the robot finds the object quickly and says, "Drawers found. Done!" in just a few words, the coach gives a "gold star."

The robot learned to "do" rather than "talk." It figured out how to get the job done using the fewest words possible. This cut the time it took to think and speak by 71.8%.

3. The "Compact Suit" (Quantization)

Finally, to make the robot even faster, they put the brain in a "compact suit." This is a technical trick called quantization, which is like compressing a high-resolution photo into a smaller file size without losing the important details. This made the robot's brain run even faster on its small hardware.

The Final Outcome

By combining these three tricks:

Learning from a Master (Distillation),
Learning to be concise (E-RLVR), and
Compressing the brain (Quantization),

The researchers created a robot that can understand complex instructions like "Find the person sitting at the table" and navigate a real apartment all by itself, without needing the internet.

In their real-world test, they sent a handheld robot into an apartment with four different missions: find a sofa, a piano, a bathtub, and a person. The robot successfully built a mental map of the rooms, found the objects, and stopped exactly where it needed to, proving that a small, local brain can do the work of a giant cloud brain.

In short: They took a super-smart but slow cloud brain, taught a small local brain how to think like it, taught the small brain to stop talking so much, and squeezed it into a tiny package. Now, the robot can think and move fast, right on the device.

Technical Summary: LocalNav

Problem Statement

Vision-Language Models (VLMs) have revolutionized embodied navigation by enabling open-vocabulary Object Goal Navigation (ObjectNav), allowing agents to locate specific, semantically complex targets (e.g., "drawers below the microwave") rather than generic categories. However, a critical bottleneck exists: state-of-the-art (SotA) VLMs are computationally prohibitive for local execution on resource-constrained robotic hardware (e.g., embedded GPUs like the Jetson Orin). Consequently, these systems are typically confined to cloud execution, introducing network latency and dependency issues that hinder low-latency, real-time robotic control.

Existing approaches face a dichotomy:

Low-Level Action Prediction (VLAs): These models predict discrete actions directly but require continuous inference (5–10 Hz), imposing a severe computational burden that is difficult to sustain on edge devices.
High-Level Decision-Making: These systems use VLMs sparsely for abstract reasoning (e.g., via Scene Graphs) while delegating low-level control to dedicated policies. While this relaxes latency constraints, SotA implementations rely on massive, cloud-bound frontier models, failing to achieve fully local deployment.

The challenge is to achieve SotA open-vocabulary ObjectNav performance on local, edge-deployable hardware without sacrificing the reasoning capabilities of large frontier models.

Methodology

The authors propose LocalNav, a deployment-first framework that integrates a modular Scene Graph (SG)-based architecture with a two-stage distillation and optimization pipeline.

1. System Architecture

The system decouples high-level reasoning from low-level control:

Scene Representation: An explicit, online-updated Scene Graph (SG) constructed from RGB-D data and odometry. It encodes rooms, objects, and exploration frontiers.
Prompting Strategy: The VLM receives a multimodal prompt consisting of:
- Text: A list of the agent's current location, explored/unexplored rooms, and observed objects with IDs.
- Image: The current RGB frame augmented with projected IDs of visible SG objects (inspired by "set-of-marks" prompting) to ground symbolic IDs in visual regions.
Action Space: The VLM selects from four high-level actions: navigate(object_id), explore(room_id), find_new_rooms(), or done(). A dedicated low-level PointGoal policy executes the physical movement.

2. Training Pipeline

To enable local execution, the authors employ a two-stage training strategy on a lightweight Qwen3.5-4B model:

Stage I: Supervised Fine-Tuning (SFT) via Distillation:
- Teacher Models: Reasoning traces are recorded from frontier cloud models (Gemini 3.1 Pro, GPT-5.4, and Claude Sonnet 4.6) performing ObjectNav in simulation.
- Data Efficiency: Only successful trajectories (approx. 500 samples per teacher) are used to fine-tune the student model. This avoids the need for millions of samples or complex feature-space alignment.
- Goal: Transfer complex spatial-semantic reasoning capabilities from the large cloud models to the 4B parameter local model.
Stage II: Embodied Reinforcement Learning from Verifiable Rewards (E-RLVR):
- Objective: Address the inference latency bottleneck caused by verbose autoregressive token generation (TG).
- Mechanism: The model is trained in a closed-loop simulation using Group Relative Policy Optimization (GRPO) with Low Rank Adaptation (LoRA).
- Reward Function ( $R_{tot}$ ): A multi-objective reward combines:
  - $R_{done}$ : Success/failure of reaching the goal.
  - $R_{nav}$ : Progress toward the goal.
  - $R_{exp}$ : Discovery of new nodes or the goal in the SG.
  - $R_{brev}$ : Token Brevity. A linear reward penalizing long output sequences, incentivizing the model to compress its reasoning without losing navigation accuracy.
- Outcome: The model learns to "think" efficiently, reducing output token length while maintaining task performance.

3. Deployment Optimization

The final system is deployed on a Jetson Orin AGX using llama.cpp. The authors apply quantization (specifically the IQ4-XS format) to further accelerate the autoregressive decoding phase, which accounts for the majority of runtime.

Key Contributions

Local VLM Optimization: Demonstrated that a 4B-parameter VLM can be optimized for ObjectNav, achieving a Success Rate (SR) of 34.5% on the HM3D OVON benchmark, narrowing the gap to cloud-based models (39.7% SR).
Distillation Analysis: Showed that distilling reasoning from frontier models using a minimal dataset (~500 traces) is sufficient to transfer complex navigation capabilities, avoiding the need for massive SFT datasets.
E-RLVR Training Strategy: Introduced a reinforcement learning approach that explicitly regularizes token generation length. This reduced output sequence length by 72.1% and inference latency by 71.8% without significantly degrading navigation performance.
Performant System Architecture: Built a modular, SG-based navigation system that supports drop-in replacement of VLMs, achieving SotA performance (39.7% SR) with cloud models and viable local deployment.

Results

Benchmark Performance:
- Cloud Baseline (Claude Sonnet 4.6): 39.7% SR, 19.7% SPL.
- Local Student (Qwen3.5-4B + SFT + E-RLVR): 34.5% SR, 17.2% SPL.
- The local model achieves a performance gap of only 5.2 percentage points in SR compared to the cloud frontier model.
Efficiency Gains:
- Token Reduction: E-RLVR reduced average output tokens per query from ~535 to ~149 (a 72.1% reduction).
- Latency: Combined with quantization, the system achieved an 82.8% reduction in overall inference latency per episode (from 305.2s to 52.5s on Jetson Orin).
- Query Frequency: The number of VLM queries per episode was reduced from ~30 (zero-shot) to ~8 (optimized).
Real-World Validation: The system was successfully deployed on a handheld robotic device in a real apartment, completing four sequential ObjectNav tasks (finding a sofa, piano, bathtub, and a person at a table) by leveraging the SG for memory and navigation.

Significance

The paper claims that LocalNav presents a viable paradigm for low-latency, high-performance embodied intelligence on physical robotic hardware. By combining modular high-level decision-making with efficient distillation and token regularization, the authors demonstrate that resource-constrained edge devices can execute complex open-vocabulary navigation tasks without relying on cloud connections. This work bridges the gap between the reasoning capabilities of frontier models and the practical constraints of on-board robotic deployment, offering a path toward truly autonomous, open-vocabulary agents.