💻 computer science

Real-Time Evaluation of Autonomous Systems under Adversarial Attacks

This paper presents an offline framework for evaluating the adversarial robustness of autonomous driving policies trained on real-world intersection data, demonstrating that architectural choices and state representations critically influence stability against gradient-based attacks despite comparable nominal accuracy.

Original authors: Adithya Mohan, Xujun Xie, Venkatesh Thirugnana Sambandham, Torsten Schön

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Adithya Mohan, Xujun Xie, Venkatesh Thirugnana Sambandham, Torsten Schön

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to drive a car through a busy city intersection. Usually, engineers test these robots in a video game (simulation) because it's safe and cheap. But this paper argues that video games are too perfect. They miss the messy, real-world glitches—like bad GPS signals, confusing road markings, or slightly wrong data—that happen when a real car is actually on the road.

The authors built a "stress test" for self-driving cars using real data collected from a real Audi Q8 in Germany. They didn't just check if the car drives well; they checked if the car breaks when someone tries to trick it.

Here is the breakdown of their experiment using simple analogies:

1. The Three "Students" (The Models)

The researchers trained three different types of AI "students" to learn how to drive through intersections. They all studied the exact same textbook (the real-world data), but they used different learning styles:

The Straightforward Student (MLP): This model is like a student who memorizes rules in a simple, linear way. It looks at the current situation and guesses the next move.
The Focused Student (Transformer): This model is like a student who uses a highlighter to pay attention to specific details (like "that red car is close" or "the lane is curving"). It breaks the road down into little "tokens" or pieces to understand the whole picture better.
The Reward-Seeker (GAIL/IRL): This model doesn't just copy the teacher; it tries to figure out why the teacher made those moves. It plays a game where it tries to fool a judge into thinking it's the expert driver.

2. The "Magic Trick" (Adversarial Attacks)

Once the students were trained, the researchers tried to trick them. They didn't change the road or the car; they changed the data the car was reading.

Think of it like this: Imagine you are driving, and someone puts a tiny, almost invisible sticker on your dashboard that slightly distorts your speedometer or your view of the lane. To you, it looks normal, but the car's computer sees a completely different reality.

The "One-Click" Trick (FGSM): A quick, single nudge to the data to see if the car gets confused.
The "Relentless" Trick (PGD): A slow, repeated series of nudges, constantly adjusting the trick to find the exact weak spot in the car's brain.

3. The Results: "Good Grades" vs. "Real-World Survival"

Here is the shocking finding: All three students got excellent grades on their final exam (clean driving). They all predicted the path perfectly when the road was normal.

However, when the "magic trick" (the adversarial attack) was applied:

They all failed spectacularly. Even though they were smart, a tiny, invisible change in the data caused them to drive wildly off course.
The "Relentless" trick was the worst. While the "One-Click" trick made the car drift a bit, the "Relentless" trick caused the car to veer off by up to 8 meters (about 26 feet). That's like driving straight into a different lane or off the road entirely.
Smarter models didn't save the day. The "Focused Student" (Transformer) and the "Reward-Seeker" (GAIL) were just as vulnerable as the "Straightforward Student." Being more complex or having a "smarter" learning style didn't make them immune to the trick.

4. The "Smooth but Wrong" Problem

The paper points out a scary detail about how these cars failed. When tricked, the car didn't just jerk around randomly. It often generated a path that looked smooth and physically possible (like a real car turning), but it was semantically wrong.

Analogy: Imagine a driver who smoothly turns the wheel to drive through a brick wall because the "wall" was tricked into looking like a "lane." The movement is smooth, but the result is a crash.

The Bottom Line

The main takeaway is that you cannot trust a self-driving car just because it drives well in normal conditions.

The paper proves that:

Simulation isn't enough: You have to test on real data to find these weaknesses.
Accuracy isn't safety: A car can be 99% accurate in normal driving but 0% safe when someone tries to trick it.
Complexity doesn't equal security: Making the AI "smarter" or more complex doesn't automatically make it harder to trick.

The researchers created a new "stress test" framework to help engineers find these hidden cracks before they put cars on the road. They found that currently, none of the standard methods are truly safe against these digital tricks, and we need to build specific defenses to stop them.

Technical Summary: Real-Time Evaluation of Autonomous Systems under Adversarial Attacks

Problem Statement

While adversarial testing has become standard in simulation for autonomous driving, purely virtual evaluations fail to capture structural inconsistencies, supervision constraints, and state-representation effects inherent in real-world data. These factors fundamentally shape policy robustness but are often abstracted away in simulation. Furthermore, while robustness vulnerabilities in reinforcement learning have been analyzed, controlled evaluations on structured real-world trajectory datasets remain limited. Existing physical-world attacks often target upstream perception pipelines (e.g., cameras, LiDAR), whereas this work addresses the vulnerability of the low-dimensional state vector itself, assuming the perception layer has been bypassed or compromised.

Methodology

Experimental Framework

The authors propose an offline trajectory-learning and adversarial robustness evaluation framework grounded in real-world intersection driving data. The evaluation is conducted in an open-loop mode on a full-scale Audi Q8 test vehicle equipped with a multi-sensor stack (LiDAR, cameras, GNSS/IMU) and an Autoware-based software framework. Trained policies perform real-time inference on live vehicle data but do not influence vehicle control, ensuring safety while allowing for realistic robustness assessment under actual sensing and synchronization conditions.

Data and State Construction

The study utilizes the UrbanIng-V2X dataset, focusing on single-ego trajectory learning across three distinct intersection scenarios in Ingolstadt, Germany.

State Representation ( $s_t$ ): A structured 97-dimensional vector is constructed for each timestamp, comprising:
- Ego dynamics (4D): Speed, acceleration, yaw rate, and heading.
- Lane-relative geometry (3D): Lateral displacement, heading error, and curvature.
- Surrounding objects (80D): Features for the $K=10$ nearest traffic participants (position, velocity, dimensions, class, distance).
- Validity mask (10D): Binary indicators for object presence.
Target: A 20-step future trajectory ( $\tau_t$ ) representing $(\Delta x, \Delta y, \Delta \psi, v)$ .

Models Evaluated

Three trajectory-learning paradigms are compared under identical state and supervision contracts:

BC-MLP: A feedforward Multi-Layer Perceptron (two hidden layers, 512 units) performing Behavior Cloning.
BC-Transformer: A Transformer-based model using object tokenization (ego, lane, and 10 object tokens) with a classification token for global aggregation.
IRL-GAIL: An Inverse Reinforcement Learning approach formulated as a Generative Adversarial Imitation Learning (GAIL) minimax game, where the policy network shares the BC-MLP architecture.

Adversarial Attack Protocol

Inference-time robustness is assessed by applying gradient-based perturbations to the structured state vector $s_t$ using:

FGSM (Fast Gradient Sign Method): Single-step perturbation.
PGD (Projected Gradient Descent): Iterative 10-step perturbation.
Constraints: Perturbations are bounded by an $\ell_\infty$ budget ( $\epsilon = 0.05$ ) applied to all state dimensions without per-feature normalization.
Metrics: Average Displacement Error (ADE) and Final Displacement Error (FDE) are computed against ground-truth trajectories.

Key Contributions

Real-World Offline Pipeline: Establishment of a structured offline trajectory-learning pipeline based on real-world intersection data, moving beyond purely simulated adversarial testing.
Controlled Paradigm Comparison: A systematic comparison of BC (MLP), BC (Transformer), and offline GAIL-style IRL under identical state representations and supervision contracts.
Systematic Robustness Evaluation: A structured evaluation matrix varying model family, attack type (Clean, FGSM, PGD), and intersection scenario (3 crossings), covering day and night conditions.
Empirical Robustness Analysis: Demonstration that comparable nominal trajectory accuracy can conceal substantially different adversarial robustness profiles.

Results

Nominal Performance

Under clean (non-adversarial) conditions, all models achieved low prediction errors, with ADE and FDE values consistently below 0.08 across all intersections. This confirms that differences observed under attack are not due to underfitting or architectural capacity limitations but rather to the sensitivity of the learned input-output mappings.

Adversarial Degradation

FGSM Attacks: Caused significant degradation, increasing ADE by approximately 1.0–1.5 meters and FDE by 1.8–2.9 meters. BC-MLP and BC-Transformer showed comparable vulnerability, suggesting that increased architectural expressiveness (attention mechanisms) does not inherently reduce susceptibility to single-step gradient attacks.
PGD Attacks: Resulted in substantially larger errors. FDE degradation frequently exceeded 5.0 meters, reaching nearly 8.0 meters in complex intersections.
- BC-MLP exhibited the largest degradation, particularly in high-curvature scenarios, indicating sensitivity to iterative amplification.
- BC-Transformer showed modest improvements in some configurations but remained highly vulnerable overall.
- IRL-GAIL did not demonstrate consistent robustness advantages; in several cases, its degradation was comparable to or worse than BC-based models.

Failure Modes

Qualitative analysis (Fig. 3) revealed that adversarially perturbed trajectories often remain kinematically smooth but are semantically incorrect, violating lane adherence or intersection rules. Errors tended to accumulate over the prediction horizon, leading to large terminal displacement errors (high $\Delta$ FDE relative to $\Delta$ ADE).

Significance and Claims

The paper claims that nominal trajectory prediction accuracy is not indicative of adversarial robustness. The study establishes that:

Architectural Inductive Biases Matter: State-structure design and architectural choices critically influence adversarial stability, leading to markedly different robustness profiles despite similar clean performance.
Training Objectives Do Not Guarantee Robustness: Adversarial imitation objectives (GAIL) and increased model complexity do not inherently confer inference-time robustness against state perturbations.
Need for Scenario-Aware Evaluation: Robustness is jointly determined by model architecture and local state geometry (e.g., curvature, merging traffic), necessitating scenario-aware evaluation rather than aggregate metrics alone.

The proposed framework serves as a scalable, reproducible benchmark for studying offline trajectory learning and adversarial robustness in real-world autonomous driving settings, highlighting the gap between clean-condition accuracy and safety-relevant behavior under attack.