💻 computer science

Development of Vision-Language Model-based GNSS Spoofing Detection for Autonomous Vehicle Navigation

This paper presents the first Vision-Language Model-based framework for detecting GNSS spoofing attacks in autonomous vehicles by fusing visual and sensor data through a three-stage fine-tuning process, achieving 94–95% F1-scores on a newly generated real-world dataset while significantly reducing computational overhead via an adaptive inference policy.

Original authors: Mohammed Aldeen, Muhammad Sami Irfan, Sagar Dasgupta, Long Cheng, Mizanur Rahman, Mashrur Chowdhury

Published 2026-07-28

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Mohammed Aldeen, Muhammad Sami Irfan, Sagar Dasgupta, Long Cheng, Mizanur Rahman, Mashrur Chowdhury

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are driving a self-driving car, a high-tech robot on wheels that relies on a digital compass called GNSS (Global Navigation Satellite System) to know exactly where it is. This compass is the car's primary sense of direction, telling it when to turn, when to stop, and how fast to go. But what if a sneaky hacker could trick that compass? They could send a fake signal that makes the car think it's turning left when it's actually going straight, or make it believe it's stopped at a red light while it's actually speeding down the highway. This is called "spoofing," and it's like a magician pulling a fast one on the car's brain.

To catch these tricks, scientists have traditionally tried to check the compass against other sensors, like a speedometer or a camera, but often in a very mathematical way—comparing numbers to numbers. The problem is that a clever hacker can sometimes make the fake numbers look perfect on paper, even if the car is doing something dangerous. This is where a new kind of AI comes in: Vision-Language Models (VLMs). Think of these as super-smart robots that can "see" a video and "read" a story at the same time. Instead of just checking if the numbers match, a VLM can understand the story of what the car is doing. It can look at the road through the camera, feel the car's movement through its sensors, and say, "Hey, the compass says we are turning right, but the video and the feeling of the car say we are turning left!" This paper explores how to use these smart AI storytellers to spot when a self-driving car is being tricked by a fake GPS signal.

The Paper's Big Idea: Teaching AI to Spot the Lie

In this study, the researchers built the very first system that uses a Vision-Language Model (VLM) to catch GPS spoofing attacks on self-driving cars. They realized that while old methods checked if the GPS numbers matched the math of the road, a smart AI could check if the GPS story matched the real story of the drive.

The Setup: A Tale of Two Cities
To train their AI, the team used a dataset of driving videos and sensor data collected in Tokyo, Japan. This was their "training school." They taught the AI to look at a short clip of a road (about 4 seconds long) and read the car's speed and turning data, then guess what the car was doing: going straight, stopping, turning left, or turning right.

But here is the tricky part: they didn't just test the AI in Tokyo. To prove the AI was actually learning the physics of driving and not just memorizing Japanese street signs, they drove a real car around Tuscaloosa, Alabama. This was their "final exam." The roads, signs, and driving habits were totally different, but the laws of physics (how a car turns or stops) remained the same.

The Three-Stage Training
The researchers didn't just dump all the data on the AI at once. They used a clever three-step training process, like teaching a student in stages:

Visual Grounding: First, they taught the AI to look at the video and understand motion (like seeing the world shift when you turn).
Sensor Calibration: Next, they taught the AI to read the numbers from the car's sensors (speed, acceleration, and turning rate) and understand what those numbers mean in plain language.
The Grand Fusion: Finally, they let the AI look at the video and the numbers together. Now, the AI could say, "The video shows the car turning right, and the numbers show a sharp turn to the right, so the car is definitely turning right."

The Attack Scenarios
To test if their system worked, the researchers created three types of "fake" attacks on the Tuscaloosa data:

The Wrong-Turn Trick: The fake GPS told the car it was turning left, but the car was actually turning right.
The Overshoot (Freeze) Trick: The car was moving forward, but the fake GPS said, "You are still here," freezing the car's position on the map.
The Stop Trick: The car was sitting still at a stop sign, but the fake GPS said, "You are moving slowly," making the car think it was driving.

The Results: A Huge Win
When they tested the system on the Tuscaloosa data, the results were striking.

The Baseline: If they used a standard, pre-trained AI without any special training, it only got about 23% to 32% of the attacks right. It was basically guessing.
The Trained AI: After their three-stage training, the AI got 94% to 95% of the maneuvers correct.
The Attack Detection:
- For the Wrong-Turn and Stop attacks, the AI caught 100% of them.
- For the Overshoot (freeze) attacks, it caught between 88% and 93%.

The "Smart Gating" Trick
There was one big problem: these smart AIs are heavy. Running them on every single second of driving would drain the car's battery and slow it down. To fix this, the team invented an "adaptive inference policy."

Think of it like a security guard who doesn't check every single car that drives by, but only checks the ones that look suspicious. The system watches the car's speed and turning rate. If the car is just cruising straight at a steady speed, the system says, "No need to call the big AI; the car is probably fine." It only wakes up the heavy AI when the car speeds up, slows down, or turns.

This trick reduced the number of times the AI had to be called by 86% (down to just 14% of the time). When the AI did run, it took about 65 to 73 milliseconds per 4-second window. This means the system is fast enough to be practical, saving a massive amount of computing power while still catching the bad guys.

What the Paper Says It's Not
The authors are very clear about what their system doesn't do. They aren't trying to fix the GPS signal itself or stop the hacker from broadcasting fake radio waves. They also aren't trying to catch "drift" attacks where the GPS slowly moves the car's position by a few inches without changing the driving behavior. Their system is specifically designed to catch attacks that change the story of the drive—like making the car think it's turning the wrong way or moving when it's stopped.

The Bottom Line
This paper shows that by teaching a smart AI to understand the story of a drive—combining what the camera sees with what the sensors feel—we can catch GPS hackers much better than before. While the system still needs powerful computers to run (it's not yet small enough for a tiny car chip), the "smart gating" trick proves that we can make this defense practical. It's a new layer of safety that doesn't just check the numbers; it checks the logic of the drive.

Technical Summary: Development of Vision-Language Model–based GNSS Spoofing Detection for Autonomous Vehicle Navigation

Problem Statement
Autonomous vehicles (AVs) rely heavily on Global Navigation Satellite Systems (GNSS) for localization and navigation, rendering them vulnerable to spoofing attacks. Unlike jamming, which denies service, spoofing attacks covertly manipulate a vehicle's perceived position and trajectory. While existing defenses include receiver-level integrity checks (e.g., RAIM), RF-level signal defenses, and multi-sensor fusion (combining GNSS with IMU, LiDAR, or odometry), these approaches face fundamental limitations. Traditional metric-based methods can be bypassed by sophisticated attacks that maintain geometric consistency with road networks or other sensor readings (e.g., the "FusionRipper" attack). Furthermore, visual-inertial odometry (VIO) and camera-to-map alignment are prone to cumulative drift and degrade under adverse visual conditions (low light, weather). Consequently, there is a need for a detection mechanism that validates GNSS not merely at the metric trajectory level, but at the semantic behavior level—verifying whether the maneuver implied by GNSS aligns with the actual driving behavior observed by the vehicle.

Methodology
The paper proposes a novel framework that fuses front-camera visual data with in-vehicle sensor readings (speed, longitudinal acceleration, and yaw rate) using a Vision-Language Model (VLM) to detect discrepancies between predicted maneuvers and GNSS-derived maneuvers.

Dataset Construction:
- Training: The model was trained on the CoVLA dataset (Tokyo, Japan), a real-world corpus containing synchronized video, GNSS, IMU, and CAN-bus data.
- Validation: An independent, real-world dataset was collected in Tuscaloosa, Alabama, using a different vehicle and camera hardware to test cross-regional generalization. This dataset was used exclusively for evaluation.
- Attack Generation: Three spoofing scenarios were simulated on the validation dataset by manipulating GNSS coordinates while keeping visual and sensor data authentic:
  - Wrong-Turn Attack: Mirroring vehicle position along the longitude axis and snapping to the road network to force an incorrect turn direction.
  - Overshoot Attack: Freezing the reported GNSS position while the vehicle continues to move.
  - Stop Attack: Injecting a slow synthetic drift in GNSS coordinates while the vehicle is stationary.
VLM Architecture and Fine-Tuning:
- Input Representation: Sensor data (speed, acceleration, yaw) is serialized into natural language text prompts describing the net change ( $\Delta$ ) over a 4-second window (12 frames at 3 fps). This creates a shared semantic space between visual tokens and sensor telemetry.
- Three-Stage Fine-Tuning: To address the lack of automotive sensor understanding in pre-trained VLMs and ensure robust cross-modal alignment, a three-stage process was employed:
  - Stage 1 (Visual Grounding): Train the vision encoder on video clips to map visual features to the LLM's semantic space.
  - Stage 2 (Sensor Calibration): Freeze the vision encoder; train the adapter and LLM to interpret the text-encoded sensor deltas.
  - Stage 3 (Joint Multimodal Fusion): Unfreeze all components and jointly fine-tune on concatenated image embeddings and sensor-delta sentences.
- Detection Logic: The fine-tuned VLM predicts the vehicle's maneuver (straight, left turn, right turn, stopped). This prediction is compared against the maneuver derived from GNSS data (via road-link transitions and bearing angles). Discrepancies trigger spoofing alerts.
Adaptive Inference Policy:
To mitigate the computational cost of running large VLMs in real-time, an adaptive gating mechanism was introduced. The system monitors speed and yaw rate changes over a sliding window. If these metrics remain stable (below defined thresholds), the system reuses the previous prediction, skipping the VLM inference. VLM invocation is triggered only when significant maneuver changes are detected or when GNSS-derived motion state contradicts in-vehicle sensor data.

Key Contributions

First VLM-Based Framework: Development of the first framework for GNSS spoofing detection in AVs that operates in a semantic behavior space, fusing visual and sensor data to detect behavioral inconsistencies.
Cross-Regional Validation: Generation of an independent real-world dataset in Tuscaloosa, Alabama, demonstrating that the model learns domain-invariant driving-behavior cues (physics-based motion patterns) rather than region-specific visual features, achieving high generalization from Tokyo training data.
Adaptive Optimization: Introduction of an inference policy that reduces VLM invocations by approximately 86% (to ~14% of total windows) while maintaining detection accuracy, addressing the latency and computational bottlenecks of large models.

Results
Evaluated on the Tuscaloosa validation dataset:

Baseline Performance: Zero-shot pre-trained VLMs (LLaVA-Next-video-7B, NVILA-8B-Video) achieved F1-scores between 23% and 32%.
Fine-Tuned Performance: The three-stage fine-tuned models achieved F1-scores between 94% and 95%.
Attack Detection Accuracy:
- Wrong-Turn Attacks: 100% accuracy for both models.
- Stop Attacks: 100% accuracy for both models.
- Overshoot Attacks: 88%–93% accuracy.
Computational Efficiency: The adaptive policy reduced total prediction time to 65ms–73ms per 4-second window (including the 14% of windows requiring VLM inference). Peak GPU memory usage was 14GB–16GB.

Significance and Claims
The paper claims to provide a practical, on-road layer of defense that complements existing signal-level integrity checks. By shifting validation from metric consistency to semantic behavior consistency, the approach addresses a gap where sophisticated spoofing attacks can pass geometric checks but imply unsafe or incorrect driving behaviors (e.g., turning the wrong way or moving when stopped). The authors emphasize that the method's reliance on physics-based motion cues (sensor deltas) and temporal visual shifts allows it to generalize across different road environments and hardware configurations.

However, the paper maintains modesty regarding immediate deployment. It acknowledges that the current implementation requires high-end server hardware (NVIDIA A100) and significant memory, making it a proof-of-concept rather than a ready-for-production solution for standard automotive edge units. The authors note that future work requires model compression (quantization, distillation) and validation against actual RF-level spoofing attacks using hardware signal generators. Additionally, the framework is not designed to detect "localization-drift" attacks that preserve high-level semantic behavior while slowly biasing position.

The Paper's Big Idea: Teaching AI to Spot the Lie

Technical Summary: Development of Vision-Language Model–based GNSS Spoofing Detection for Autonomous Vehicle Navigation

More like this