Temporal Misalignment Attacks against Multimodal Perception in Autonomous Driving

This paper introduces DejaVu, a novel attack that exploits in-vehicular network vulnerabilities to induce subtle temporal misalignments between camera and LiDAR streams, thereby severely degrading multimodal fusion-based perception tasks like object detection and tracking in autonomous driving systems.

The Gist

Imagine a self-driving car as a highly skilled conductor leading an orchestra. To navigate the road safely, this conductor relies on two main musicians: a Camera (the eyes that see colors, textures, and signs) and a LiDAR (a laser scanner that measures exact distances and shapes in 3D).

For the music to sound perfect, these two musicians must play in perfect sync. If the camera sees a stop sign now, the LiDAR must measure the distance to that sign at that exact same moment. This synchronization is called Multimodal Fusion.

The paper you shared, titled "DEJAVU," reveals a terrifying new way to break this orchestra without breaking a single instrument.

The Core Problem: The "Ghost" in the Timing

In a perfect world, the camera and LiDAR send their data to the car's brain at the exact same time. But in reality, they are slightly out of sync. The car's software has a "tolerance window"—a tiny buffer that says, "If the camera and LiDAR data arrive within 0.1 seconds of each other, let's assume they are looking at the same thing."

The DEJAVU attack is a hacker who doesn't try to blind the camera or jam the LiDAR. Instead, they play a trick on the clock.

The Attack: The "Time Travel" Trick

Imagine you are watching a live sports game with a friend. You both have stopwatches.

• The Camera is you.
• The LiDAR is your friend.
• The Car's Brain is the referee trying to combine your notes.

The hacker sneaks into your friend's (LiDAR's) stopwatch and secretly turns the time backwards by a few seconds.

• The camera sees a car approaching now.
• The LiDAR, however, is reporting data from 5 seconds ago (when the car was far away).
• Because the hacker also faked the timestamp on the LiDAR's message to look like it arrived "on time," the car's brain thinks, "Oh, the camera and LiDAR are in sync!"

The Result: The car fuses a "now" image with a "past" measurement. It's like trying to catch a ball that was thrown 5 seconds ago while looking at where it is right now. The math breaks, and the car gets confused.

What Happens When the Clock is Broken?

The researchers tested this on two different "jobs" the car needs to do, and found that the car is surprisingly fragile in specific ways:

1. The "Spotter" (Object Detection)

• Job: "Is there a car, pedestrian, or bike in front of us?"
• Weakness: This job relies heavily on the LiDAR (the laser ruler).
• The Attack: If the hacker delays the LiDAR by just one frame (a tiny fraction of a second), the car's ability to spot cars drops by a massive 88%.
• Analogy: It's like trying to measure the width of a door using a ruler that is stuck in the past. The car might think a pedestrian is 10 feet away when they are actually right in front of the bumper.

2. The "Chaser" (Object Tracking)

• Job: "Follow that red car as it moves down the street."
• Weakness: This job relies heavily on the Camera (the eyes).
• The Attack: If the hacker delays the camera feed by just three frames, the car loses track of the object 73% of the time.
• Analogy: It's like trying to follow a dancer while wearing glasses that show you a video of them from 3 seconds ago. You'll reach out to grab their hand, but they'll have already moved.

The Real-World Nightmare

The researchers didn't just run this on a computer; they built a test car network and simulated a full self-driving system (using a popular software called Autoware). The results were scary:

• Phantom Braking: The car sees a truck that passed by 5 seconds ago (because of the delayed data) and slams on the brakes for a ghost that isn't there. This could cause a rear-end collision.
• The Invisible Truck: The car sees a truck coming toward it in the camera feed, but the LiDAR says "nothing is there" (because it's looking at the past). The car decides the road is clear and drives straight into the truck.

Why Is This So Hard to Stop?

The paper points out that the "clock" in these cars is the weak link.

• The Trust Issue: The car assumes that if a message says "I was captured at 10:00:01," it actually was. It doesn't check if the clock was tampered with.
• The Network: Modern cars use a high-speed network (Automotive Ethernet) that is great for speed but, in many current setups, doesn't have strong security checks to verify if the time on a message is real or forged.

The Solution?

The authors suggest a few ways to fix this "time travel" vulnerability:

1. Secure Clocks: Use hardware that can't be easily hacked to change the time.
2. Double-Check: Don't just trust the timestamp. If the camera says "Car is close" but the LiDAR (even with a slight delay) says "Car is far," the car should realize something is wrong and slow down.
3. Cross-Reference: Use other sensors (like the car's speedometer or GPS) to see if the timing makes sense.

The Bottom Line

The DEJAVU attack shows that self-driving cars are incredibly smart, but they are also incredibly trusting of their own clocks. By simply messing with the time, a hacker can make a super-intelligent car see ghosts, miss real dangers, or crash into things it should have seen coming. It's a reminder that in the world of autonomous driving, time is just as important as vision.

Technical Summary

1. Problem Statement

Autonomous Driving (AD) systems rely heavily on Multimodal Fusion (MMF), primarily combining Camera and LiDAR data, to achieve robust scene understanding. While these sensors have complementary strengths (e.g., LiDAR for depth, Cameras for texture), they operate at different sampling rates and are inherently asynchronous. To fuse them, systems rely on temporal alignment, pairing data streams based on timestamps.

The paper identifies a critical vulnerability: Temporal Misalignment.

• Current Assumptions: AD systems assume that:
  1. Global clock synchronization (via gPTP/TSN) is secure.
  2. Sensor ECUs provide accurate, trusted timestamps.
  3. Middleware (e.g., ROS2/DDS) ensures data integrity and freshness.
• The Threat: An adversary can exploit vulnerabilities in in-vehicle networks (e.g., OBD-II, OTA updates, or unsecured ROS2 nodes) to manipulate timestamps without altering the actual sensor data payload. By injecting subtle delays or clock drifts, an attacker can force the fusion algorithm to pair semantically inconsistent data (e.g., a LiDAR scan from $t-1$ with a Camera frame from $t$). This "semantic misalignment" degrades perception accuracy, leading to safety-critical failures like missed detections or phantom braking.

2. Methodology: The DEJAVU Attack

The authors propose DEJAVU, a temporal misalignment attack that exploits the fragility of MMF systems.

Attack Mechanism

The attack manipulates the Universal Clock Offset (UCO) or the Timestamp Integrity of sensor messages.

• Goal: Create a scenario where the reported timestamp difference between sensors is within the system's tolerance window ($\tau$), but the actual physical capture time difference is large.
• Execution:
  1. Clock Synchronization Disruption (Capability C1): The attacker impersonates a Grandmaster clock in the PTP network, forcing slave nodes (sensors) to drift or step backward in time.
  2. Timestamp Manipulation (Capability C2): The attacker directly modifies the timestamp header of packets while keeping the payload (point cloud/image) unchanged.
  3. Node Impersonation (Capability C3): In ROS2 environments, the attacker impersonates a sensor publisher, replaying old data with new timestamps to pollute the fusion queue.

Attack Strategies

• Constant Delay: Applies a fixed offset to all messages, causing a systematic lag in perception.
• Random Delay: Applies stochastic delays, disrupting the temporal sequence crucial for tracking.
• Unimodal vs. Multimodal: Attacks can target a single sensor (e.g., only LiDAR) or both simultaneously.

3. Key Contributions

1. Novel Attack Vector: Introduction of DEJAVU, the first comprehensive study demonstrating how temporal misalignment attacks can severely degrade MMF-based perception in AD.
2. Modality-Specific Vulnerability Discovery: The authors reveal a critical asymmetry in how perception tasks rely on different sensors:
   • 3D Object Detection: Highly dependent on LiDAR. Even a single-frame LiDAR delay causes catastrophic failure.
   • Multi-Object Tracking (MOT): Highly dependent on Camera inputs. Camera delays cause significant tracking degradation (identity switches).
3. Hardware-in-the-Loop (HIL) Validation: Realized the attack on an Automotive Ethernet testbed using Raspberry Pi nodes and TSN, proving feasibility in a controlled network environment.
4. End-to-End Simulation: Integrated DEJAVU into the Autoware stack (a production-grade AD simulator) to demonstrate real-world consequences, including collisions and phantom braking.
5. Defense Analysis: Proposed and evaluated a One-Class SVM (OC-SVM) based detector, highlighting the difficulty of detecting random temporal attacks compared to constant ones.

4. Evaluation Results

The authors evaluated DEJAVU on state-of-the-art models (MVXNet, BEVFusion, MMF-JDT) using the KITTI and nuScenes datasets.

A. Impact on 3D Object Detection

• LiDAR Sensitivity: Detection models are extremely sensitive to LiDAR delays.
  • MVXNet (KITTI): A 1-frame LiDAR delay reduced the mean Average Precision (mAP) for cars from 84.1% to 9.7% (an 88.5% drop).
  • BEVFusion (nuScenes): A 1-frame LiDAR delay reduced car mAP by 65.6%.
• Camera Resilience: Delays in the camera stream had negligible impact on detection performance (mAP remained stable).
• Compounding Effect: When both sensors were delayed simultaneously, the performance drop was even more severe (e.g., BEVFusion car mAP dropped by 89.2% with 1-frame delay on both).

B. Impact on Multi-Object Tracking (MOT)

• Camera Sensitivity: Tracking models rely heavily on visual texture and temporal continuity.
  • MMF-JDT (KITTI): A 3-frame camera delay caused the Multiple Object Tracking Accuracy (MOTA) to drop by 73%.
  • Identity Switches (IDSW): Camera delays led to a dramatic increase in ID switches, breaking object continuity.
• LiDAR Resilience: Tracking performance was less affected by LiDAR delays compared to camera delays.

C. End-to-End Simulation (Autoware)

In a full-stack simulation:

• False Negatives: Delayed LiDAR data caused the vehicle to miss an oncoming truck, resulting in a simulated head-on collision.
• False Positives: Delayed data caused the vehicle to perceive a truck that had already passed, triggering unnecessary emergency braking (phantom braking).
• Localization Drift: Temporal inconsistencies desynchronized SLAM modules, causing the vehicle to veer off-lane.

5. Significance and Implications

• Security Gap: The paper exposes a fundamental gap in AD security: while sensor spoofing (modifying data) is well-studied, temporal integrity is often overlooked. Current defenses (like consistency checks) often fail to detect attacks where the data is "real" but the timing is wrong.
• Architectural Flaw: The results suggest that current MMF architectures are not robust to temporal inconsistencies. They rely too heavily on a single modality (LiDAR for detection, Camera for tracking) rather than true redundancy.
• Safety Critical: The attack is stealthy (payloads are unmodified) and can lead to life-threatening scenarios (collisions) without triggering standard anomaly detectors.
• Future Directions: The paper advocates for:
  • Hardening: Cryptographic authentication of timestamps and hardware-anchored clocks.
  • Detection: Cross-modal consistency checks and kinematic verification (using IMU/CAN data) to detect temporal drift.
  • Mitigation: Adaptive fusion strategies that down-weight sensors with low temporal confidence.

In conclusion, DEJAVU demonstrates that the "time" dimension in autonomous driving is as critical as the "space" dimension, and compromising temporal synchronization can effectively blind or confuse autonomous vehicles.