💻 computer science

PanguMotion: Continuous Driving Motion Forecasting with Pangu Transformers

This paper introduces PanguMotion, a novel motion forecasting framework for autonomous driving that leverages Pangu-1B Transformer blocks to enhance feature representation and addresses the limitations of existing methods by modeling continuous driving scenarios through a reorganized Argoverse 2 dataset.

Original authors: Quanhao Ren, Yicheng Li, Nan Song

Published 2026-03-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Quanhao Ren, Yicheng Li, Nan Song

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to drive a car. The hardest part isn't just seeing the road right now; it's predicting what will happen next. Will that pedestrian step off the curb? Will the car in front brake suddenly?

Most current self-driving AI systems are like amnesiac tourists. They look at a photo of the street, make a guess about the next second, and then immediately forget everything. They treat every moment as a brand-new, isolated snapshot.

PanguMotion is a new approach that changes the game. It's like giving that robot a photographic memory and a super-intelligent co-pilot who has read every book in the library.

Here is the breakdown of how it works, using simple analogies:

1. The Problem: The "Snapshot" vs. The "Movie"

Old Way (The Snapshot): Imagine watching a movie where every 5 seconds, the screen goes black, and you have to guess what happens next based only on the single frame you just saw. You have no idea if the car was speeding up or slowing down. This is how most self-driving AI works today.
The RealMotion Fix: A previous invention called "RealMotion" fixed this by stitching those snapshots together into a continuous movie. It remembers the history of the drive, so the AI knows the car was accelerating, not just "there."
The PanguMotion Upgrade: This paper takes that "continuous movie" idea and adds a genius editor to the mix.

2. The Secret Ingredient: The "Frozen Brain"

The researchers took a massive, pre-trained Large Language Model (LLM) called Pangu-1B.

What is Pangu? Think of Pangu as a super-smart librarian who has read billions of books (mostly in Chinese). It understands patterns, relationships, and the "flow" of stories incredibly well.
The Twist: Usually, we use these models to write poems or answer questions. But this paper asks: "Can this librarian help us predict traffic?"
The "Frozen" Trick: They didn't teach the librarian new things (which would take forever and cost a fortune). Instead, they "froze" the librarian's brain. They took one specific layer of this massive brain and inserted it into the driving AI.
The Analogy: Imagine your driving AI is a student taking a test. The "Frozen Pangu Block" is like having a genius tutor sitting right next to them. The tutor doesn't write the answers for them; they just look at the student's notes and say, "Hey, look at this pattern here. You missed a subtle clue. Let me highlight it for you."

3. The "Feature Enhancer" (The Magic Filter)

The paper calls this a "Feature Enhancer." Here is how it works in plain English:

The driving AI looks at the road and sees thousands of data points (lines, dots, speeds). It's like looking at a messy pile of puzzle pieces.
The Pangu Block acts as a smart filter. It looks at that messy pile and instantly knows which pieces are important and which are noise. It amplifies the "good" information (like a car braking) and ignores the "bad" information (like a shadow on the road).
Because the Pangu model was trained on so much human language, it is surprisingly good at understanding sequences and relationships, which is exactly what driving is.

4. The Surprising Discovery: "Less is More"

Here is the most interesting part of the paper.

The original "RealMotion" system had two main parts: one to look at the road context (the movie) and one to track agent trajectories (the specific paths of other cars).
When the researchers added the "Genius Tutor" (Pangu), they found that the "Genius Tutor" was so good at refining the path predictions that the second part (tracking specific paths) became redundant.
The Result: They removed the second part entirely.
The Analogy: Imagine you have a team of two editors checking a manuscript. You hire a world-famous literary critic (Pangu) to review it. You realize the second editor is now just repeating what the critic says. So, you fire the second editor. Surprisingly, the book comes out better and is faster to produce because the team is less cluttered.

5. Why This Matters (The "Real World" Impact)

Better Safety: The system predicts the single most likely path of a car with much higher accuracy (about 1.5% to 2% better). In self-driving, that tiny percentage difference is the difference between a near-miss and a crash.
Hardware Ready: They didn't just do this on a powerful computer; they adapted it to run on Ascend NPUs (a specific type of Chinese AI chip). This means it's ready to be put into actual cars, not just research labs.
Cross-Domain Magic: It proves that a model trained to understand language can also understand movement. It's like proving that a person who is great at writing poetry is also great at choreographing a dance.

Summary

PanguMotion is a self-driving system that:

Remembers the past (unlike older systems).
Uses a "frozen" super-intelligent brain (Pangu) to highlight the most important details in the traffic scene.
Cuts out unnecessary parts of its own brain to work faster and more accurately.

It's a step toward cars that don't just "see" the road, but truly understand the story of the traffic around them.

1. Problem Statement

Autonomous driving motion forecasting aims to predict the future trajectories of surrounding agents to ensure safety. However, existing methods face two primary limitations:

Discrete Scene Processing: Most state-of-the-art models (e.g., Wayformer, Scene Transformer) treat driving scenes as independent, isolated snapshots. This ignores the temporal continuity and historical context inherent in real-world driving, where scenes evolve sequentially.
Underutilization of Large Models: While Large Language Models (LLMs) like Pangu have shown remarkable "information filtering" capabilities in visual tasks, their potential to enhance continuous driving forecasting frameworks remains largely unexplored.
Hardware Constraints: Many advanced models lack optimized implementations for specific hardware accelerators like Huawei's Ascend NPUs, hindering real-time deployment.

2. Methodology: PanguMotion

The authors propose PanguMotion, a framework that integrates a frozen Pangu-1B Transformer block into a continuous motion forecasting architecture. The approach is built upon the RealMotion framework but introduces significant architectural modifications.

A. Data Strategy: RealMotion Reorganization

The model utilizes the RealMotion data reorganization strategy, which transforms independent scenes from the Argoverse 2 dataset into continuous sequences. This mimics real-world driving by allowing the model to utilize historical context from successive scenes.

B. Architectural Design

PanguMotion adopts an encoder-decoder structure with a specific focus on feature enhancement:

Input Representation: Historical agent trajectories and map elements are encoded using a base encoder (PointNet-like for maps, neighborhood attention for agents).
Scene Context Stream: A cross-attention mechanism fuses current scene features with historical memory features.
Pangu Feature Enhancer (Core Innovation):
- A frozen Transformer block (specifically Layer 25) from the pre-trained Pangu-1B model is inserted after the scene encoding layers and before the decoder.
- Dimension Alignment: Linear projection layers ( $W_{before}$ and $W_{after}$ ) are added to align the model's 128-dimensional scene features with Pangu's 1536-dimensional hidden space.
- Masking Adaptation: Autoregressive masks (used for text generation) are removed; only valid token masks are retained to allow simultaneous processing of all scene tokens.
- Freezing: All Pangu parameters remain frozen during training to leverage pre-trained representations without overfitting.
Architectural Simplification: Unlike the original RealMotion dual-stream design, PanguMotion removes the agent trajectory stream. The authors hypothesize that the Pangu block's feature enhancement capability overlaps with the trajectory stream's refinement function, and removing the latter reduces redundancy.
Decoder: A multimodal projection layer expands features into 6 modes, followed by trajectory regression and probability prediction branches.

C. Hardware Adaptation

The framework includes a complete adaptation for Huawei Ascend NPU:

Replaced GPU-dependent attention mechanisms (e.g., NeighborhoodAttention1D) with NPU-compatible standard attention.
Implemented distributed data parallel training using PyTorch Lightning on Ascend 910B2 NPUs.

3. Key Contributions

Methodological Innovation: First successful integration of a Pangu large language model into a continuous driving prediction framework, demonstrating its efficacy as a feature enhancer for spatiotemporal tasks.
Architectural Optimization: Discovery that removing the agent trajectory stream in conjunction with adding the Pangu block improves single-mode prediction accuracy, suggesting functional redundancy between the two components.
Practical Feasibility: Provided a complete end-to-end training and inference solution optimized for Ascend NPUs, maintaining low latency while enhancing feature representation.

4. Experimental Results

Experiments were conducted on the Argoverse 2 dataset using the continuous sequence strategy.

Performance Metrics

Compared to the baseline RealMotion, PanguMotion achieved significant improvements in single-mode prediction (critical for decision-making):

minADE1 (Average Displacement Error): Improved by 1.58% (1.646 $\to$ 1.620).
minFDE1 (Final Displacement Error): Improved by 1.29% (4.100 $\to$ 4.047).

Ablation Studies & Analysis

Model Comparison: PanguMotion outperformed a similar architecture using LLaMA (LlamaMotion), with minADE1 improving by 5.71%. The authors suggest Pangu's pre-training on Chinese corpora may encode spatial/temporal patterns differently or more effectively for this task.
Layer Selection: Using only the last layer (Layer 25) of Pangu yielded the best results. Using multiple layers (3 or 6) degraded performance, supporting the "information filtering hypothesis" that higher layers provide the most suitable semantic representations for transfer.
Insertion Position: Inserting the block at the final layer (Layer 25) was superior to intermediate layers, confirming that high-level semantic features are most beneficial for refining scene representations.
Stream Removal: Removing the agent trajectory stream (while keeping the Pangu block) improved single-mode metrics by ~2.6% compared to keeping both streams, indicating the Pangu block effectively supersedes the trajectory stream's refinement role.

5. Significance and Impact

Cross-Domain Transferability: Validates that large language models can generalize to non-linguistic, sequential tasks like motion forecasting, acting as powerful "universal feature enhancers."
Efficiency vs. Accuracy: Demonstrates that powerful pre-trained modules allow for the simplification of complex architectures (removing the trajectory stream) while actually improving performance in the most critical metric (single-mode prediction).
Deployment Readiness: By providing a fully optimized Ascend NPU pipeline, the work bridges the gap between theoretical LLM integration and practical, real-time autonomous driving deployment on Chinese hardware ecosystems.
Open Source: The authors open-sourced the code, establishing a new baseline for continuous motion forecasting research.

Limitations & Future Work

Computational Cost: Pangu Transformers remain computationally heavy, posing challenges for resource-constrained edge devices.
Multi-Agent Complexity: The current model focuses primarily on single-agent prediction with simplified multi-agent interaction modeling.
Data Bias: The Pangu model's Chinese pre-training corpus may introduce cultural or linguistic biases affecting spatial understanding.
Real-World Gap: While data is reorganized to be continuous, it still differs from true real-world continuous driving scenarios.