🤖 AI

PGN: Design and Implementation of a Vision-Language Navigation System Based on Pangu Multimodal Foundation Model

This paper presents PGN, an offline Vision-Language Navigation system built on the OpenPangu-7B foundation model that employs a two-stage training strategy to align visual and language modalities and adapt to expert trajectories, achieving a 62.29% Normalized Action Match on held-out data while utilizing mixed-precision computation and DeepSpeed on Ascend 910B hardware.

Original authors: Li Xian, Mingxi Li, Yizheng Wang, Yiming Shen, Qi Chen, Zhuoling Xiao

Published 2026-07-21

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Li Xian, Mingxi Li, Yizheng Wang, Yiming Shen, Qi Chen, Zhuoling Xiao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where robots don't just follow rigid, pre-programmed lists of commands, but can actually understand you like a human friend. This is the dream of "embodied AI"—giving a machine a body (or a camera and wheels) and a brain that can see the room, hear your voice, and figure out how to move around. The big challenge is connecting two very different languages: the visual world of pixels and colors, and the linguistic world of sentences and instructions. Think of it like trying to teach a dog to navigate a house using only a map written in poetry. You need a way to translate "walk past the kitchen" into a series of physical turns and steps. This is the core puzzle of Vision-Language Navigation (VLN). It's not just about recognizing objects; it's about understanding the story of a room over time, remembering where you've been, and deciding what to do next based on a simple sentence.

Enter PGN, a new system built by researchers at the University of Electronic Science and Technology of China, designed to solve this puzzle using a powerful "brain" called OpenPangu-7B. You can think of OpenPangu as a super-smart, pre-trained language model that already knows how to speak and write, but it doesn't know how to see or move. The researchers' goal was to teach this language giant how to navigate a virtual house by showing it pictures and instructions. They didn't just throw the robot into the deep end; they built a two-step training camp. First, they taught the robot how to "see" by connecting its language brain to a specialized camera eye (a vision encoder) using a translator module called a Q-Former. This allowed the robot to understand that a picture of a vase is related to the word "vase."

Once the robot could understand the connection between images and words, the second stage began: learning to move. The team fed the system thousands of examples of perfect navigation paths taken by an "expert" (a computer program that never makes mistakes). The robot learned to look at a sequence of five recent snapshots of the room, read the instruction, and then—crucially—think before it acted. Instead of just blurting out "turn left," the model was trained to generate a little bit of reasoning text first, like a human saying, "Okay, I see the kitchen on my left, so I should turn left," before committing to the action. They tested this system on 500 hidden test paths where the robot was given the correct history of what it had seen, and asked to predict the next move. The results were promising: in the latest version (V9), the robot matched the expert's action 62.29% of the time and almost always gave a non-empty answer (100% of the time). However, the authors are very careful to note that this is a "teacher-forced" test. It's like a student taking a multiple-choice quiz where the teacher holds up the correct answer key at every step. The robot hasn't actually been tested on whether it can recover if it makes a mistake or if it can successfully reach a goal in a real, messy environment without help. While the system shows that a large language model can be adapted to understand navigation instructions and align with expert actions, the paper concludes that the true test of whether this robot can actually navigate a house on its own is still a job for the future.

Technical Summary: PGN (Pangu Navigator)

Problem Statement

Vision-Language Navigation (VLN) requires an embodied agent to interpret natural language instructions and predict a sequence of actions based on temporally ordered visual observations. While Multimodal Large Language Models (MLLMs) offer broad visual-language representations, adapting them to continuous navigation presents specific challenges:

Visual-Language Alignment: Bridging the gap between visual features and the language model's embedding space.
Temporal Representation: Efficiently encoding a sequence of observations without overwhelming the context window.
Action-Space Grounding: Constraining generated text to a small, discrete action space (forward, left, right, stop).
Hardware Constraints: Maintaining stable training on specific hardware (Ascend NPUs) where mixed-precision operations may be unstable.

Existing benchmarks often rely on discrete graphs or continuous environments, but adapting a 7B-scale foundation model like OpenPangu-7B to these tasks requires a specialized architecture and training pipeline that addresses these constraints simultaneously.

Methodology: PGN (Pangu Navigator)

The authors propose PGN, an offline VLN action-prediction system built on OpenPangu-7B. The system employs a two-stage training pipeline and a specific input/output structure designed for temporal reasoning and action generation.

1. Model Architecture

Visual Encoder: Uses a frozen EVA-ViT-G/14 to extract patch-level features from $224 \times 224$ observation images.
Visual-Language Alignment (PGMM): A Q-Former with 32 learnable query tokens uses cross-attention to produce a fixed-size visual representation. This is followed by a two-layer MLP projector that maps 768-dimensional query representations to the 4096-dimensional embedding space of OpenPangu-7B.
Language Backbone: OpenPangu-7B serves as the core language model.
Adaptation: LoRA (Low-Rank Adaptation) adapters are injected into the language backbone to enable task-specific learning without full fine-tuning.
Structural Tokens: Three learned tokens mark image starts, image ends, and frame boundaries to organize the multimodal input.

2. Input and Output Formulation

Input: At time step $t$ , the model receives a navigation instruction $I$ and a window of five temporally ordered observations ( $O_t = \{o_{t-4s}, \dots, o_t\}$ ), where $s$ is the sampling stride.
Output Format: The model generates reasoning text ( $r_t$ $r_{t}$ ) followed by a canonical action ( $a_t$ $a_{t}$ ). The action space is discrete: $\{forward, left, right, stop\}$ ${f or w a r d, l e f t, r i g h t, s t o p}$ .
- Factorization: $P(r_t, a_t | I, O_t) = P(r_t | I, O_t)P(a_t | r_t, I, O_t)$ .

3. Training Pipeline

Stage 1: Visual-Language Alignment (PGMM):
- Data: Image-caption pairs from Microsoft COCO and Visual Genome.
- Freezing: The vision encoder (EVA-ViT-G/14) and language backbone (OpenPangu-7B) are frozen.
- Training: Only the Q-Former and the two-layer MLP projector are optimized to align visual inputs with the language model.
Stage 2: Navigation Adaptation:
- Data: Expert navigation trajectories collected via a DAgger-compatible pipeline (using $\beta=1$ , effectively expert-only data collection followed by supervised behavior cloning).
- Freezing: The aligned visual pathway (EVA, Q-Former, projector) and the base language model parameters are frozen.
- Training: Updates are confined to the three structural-token embeddings and the LoRA adapters.
- Sampling: Uses a five-observation window with epoch-dependent temporal sampling ( $istart = e \mod s$ ) and action-aware sampling to balance forward, turning, and stopping actions.

4. Implementation Details

Hardware: Trained on eight Ascend 910B NPUs.
Precision: Utilizes mixed-precision computation with DeepSpeed ZeRO-2. To handle numerical overflow in FP16 attention-mask operations, the system employs selective FP32 computation for specific unstable operations rather than global FP32 training.

Key Contributions

Two-Stage System: A pipeline that first aligns visual and language representations (PGMM) and subsequently adapts the aligned model to expert navigation trajectories using temporal observations and LoRA.
Structured I/O Format: A novel input/output design combining five-frame observation windows, temporal boundary tokens, reasoning text generation, and a canonical four-action space.
Ascend-Oriented Implementation: A robust training stack for Ascend NPUs, featuring selective FP32 computation for stability and a teacher-forced, open-loop evaluation protocol.

Results

The system was evaluated on 500 held-out expert trajectories using a teacher-forced, open-loop protocol (where the model predicts actions based on expert observation history without executing them to generate new observations).

Metrics:
- Normalized Action Match (NAM): Measures the alignment of the predicted action with the expert action after normalizing text expressions.
- Non-empty Rate (NER): The fraction of predictions producing a non-empty response.
- Exact-string Match (EM): Strict string equality (noted as 0% for later versions due to the inclusion of reasoning text).
Performance:
- Version V9: Achieved a 62.29% NAM and 100.00% NER.
- Version V8: Achieved 62.13% NAM and 100.00% NER.
- Trend: Earlier versions (V5, V6) showed lower NAM (25.40% and 23.18%) and NER (33.98% and 99.17%), indicating that the introduction of reasoning text, identity prompting, and improved sampling in V8/V9 significantly improved action alignment.

Significance and Limitations (Author's Claims)

The paper positions PGN as a proof of concept for adapting the OpenPangu-7B foundation model to offline VLN tasks on Ascend hardware.

Feasibility: The results (62.29% NAM) support the feasibility of using a frozen-backbone, LoRA-adapted approach for offline expert-action alignment.
Scope of Evaluation: The authors explicitly state that these metrics quantify offline expert-action alignment rather than closed-loop navigation success.
- The evaluation does not measure error accumulation, path efficiency, goal completion, or the ability to recover from incorrect actions.
- Metrics like Success Rate, SPL, or navigation error are not reported.
Future Work: The authors note that evaluating the system in a closed-loop environment (executing predicted actions in the simulator) is required to establish true navigation success. The current work serves as a development record for model alignment rather than a comparison with state-of-the-art closed-loop VLN systems.