PLM-Net: Perception Latency Mitigation Network for Vision-Based Lateral Control of Autonomous Vehicles

This paper introduces PLM-Net, a modular deep learning framework that mitigates the adverse effects of perception latency on vision-based lateral control by interpolating between a frozen base controller and a time-aware prediction model, achieving significant reductions in steering error for both constant and time-varying latency scenarios.

The Gist

Imagine you are driving a car, but there's a strange glitch: every time you look at the road, turn the wheel, or press the gas, your brain and the car's steering wheel are out of sync. You see a curve coming up, but by the time your hand actually turns the wheel, you've already passed the curve. You try to correct, but you're reacting to a ghost of the past, not the reality of the present. This is perception latency.

In the world of self-driving cars, this "lag" is a huge problem. If a computer sees a curve but takes 0.2 seconds to process it and turn the wheel, the car might overshoot the turn, swerve, or even crash.

This paper introduces a clever solution called PLM-Net (Perception Latency Mitigation Network). Here is how it works, explained through simple analogies:

The Problem: The "Slow-Motion" Driver

Think of a self-driving car's brain as a driver.

• Normal Driving: The driver sees a curve, thinks "I need to turn left," and turns the wheel instantly.
• With Latency: The driver sees the curve, but their brain is stuck in "slow motion." By the time they finally turn the wheel, the car has already drifted too far. They try to over-correct, then over-correct again, leading to a zigzag mess.

Usually, engineers try to fix this by making the computer faster (buying better chips). But the authors say, "What if we can't make the computer faster? What if the lag is just part of the system?"

The Solution: The "Crystal Ball" Co-Pilot

Instead of trying to speed up the computer, PLM-Net adds a smart Co-Pilot who has a crystal ball.

1. The Base Driver (The "Base Model"): This is the original self-driving software. It's good, but it suffers from the lag. It sees the road now but acts on it later.
2. The Crystal Ball (The "TAPM"): This is the new addition. It doesn't just look at the road now; it looks at the road and predicts: "If we wait 0.1 seconds, what will the car need to do? If we wait 0.2 seconds, what will it need to do?"

How It Works: The "Time-Traveling" Steering Wheel

Imagine the Co-Pilot has a set of five different steering instructions ready, like a menu:

• Option A: Turn left now (0 seconds delay).
• Option B: Turn left for a 0.15-second delay.
• Option C: Turn left for a 0.20-second delay.
• Option D: Turn left for a 0.25-second delay.
• Option E: Turn left for a 0.30-second delay.

The system constantly checks the clock.

• If the lag is exactly 0.20 seconds, the Co-Pilot grabs Option C and tells the car to turn.
• If the lag suddenly jumps to 0.22 seconds (maybe the computer got busy processing a cloud), the Co-Pilot doesn't panic. It simply mixes Option C and Option D to create a perfect "0.22-second" turn.

It's like a DJ mixing two songs together to match the beat perfectly, no matter how the tempo changes.

Why This Is a Big Deal

Most previous solutions tried to either:

1. Rewrite the driver: Change the original software to be "lag-aware" (which is hard and risky).
2. Speed up the hardware: Buy expensive, faster computers (which isn't always possible in real cars).

PLM-Net is different. It's a "Plug-and-Play" upgrade.

• You take the existing self-driving car software (the "Base Model").
• You plug in this new "Crystal Ball" module.
• The original software doesn't even know it's there; it just keeps doing its job.
• The new module intercepts the command, checks the lag, and tweaks the steering angle just enough to compensate for the delay.

The Results: From Zigzag to Smooth Sailing

The researchers tested this in a computer simulation (a video game world for cars).

• Without the fix: When they added a 0.2-second delay, the car started swerving wildly, missing turns, and driving like a drunk driver.
• With PLM-Net: The car drove smoothly, almost exactly like it was driving with zero delay.

They found that the new system reduced steering errors by 62% for constant delays and a massive 78% for unpredictable, changing delays.

The Bottom Line

Think of PLM-Net as a smart translator for self-driving cars. When the car's brain is slow to react, this system translates the "slow" commands into "fast" actions, predicting the future so the car doesn't have to react to the past. It allows self-driving cars to stay safe and steady, even when their computers are running a bit slow.

Technical Summary

1. Problem Statement

Autonomous Vehicle (AV) control relies on a perception–planning–control cycle. A critical challenge in this cycle is perception latency ($\delta$), defined as the time delay between visual sensing and the application of a steering actuation.

• Components: Latency consists of algorithmic latency (time to infer action from observation, which varies based on scene complexity) and actuator latency (steering lag, often constant).
• Impact: Even small delays cause the vehicle to act on outdated observations. In lateral control (lane keeping), this leads to incorrect steering decisions, resulting in trajectory deviations, zigzagging, and potential lane departure, especially at higher speeds or during curves.
• Limitations of Existing Solutions:
  • Classical control methods (e.g., Model Predictive Control) often require explicit vehicle dynamics modeling and controller redesign, which is difficult for data-driven imitation learning systems.
  • Reducing latency via hardware (GPUs/FPGAs) is often impractical or insufficient for automotive platforms.
  • Previous deep learning approaches either assumed constant latency or required manual tuning of weights, lacking adaptability to time-varying delays.

2. Methodology: PLM-Net

The authors propose PLM-Net (Perception Latency Mitigation Network), a modular, plug-in deep learning framework designed to mitigate latency effects without modifying the original Base Model (BM) controller.

Core Architecture

PLM-Net consists of two main components working in tandem:

1. Base Model (BM): A pre-trained, frozen vision-based imitation learning controller (inspired by NVIDIA PilotNet) that maps visual observations ($o_t$) and vehicle speed ($v_t$) to a nominal steering action ($a^{BM}_t$).
2. Timed Action Prediction Model (TAPM): A predictive module that forecasts future steering actions corresponding to specific discrete latency values.
   • Input: It takes the BM's output action ($a^{BM}_t$) and intermediate feature vectors (image features $z^o_t$ and velocity features $z^v_t$) from the BM.
   • Structure: Inspired by Branched Conditional Imitation Learning (BCIL), the TAPM employs multiple sub-models (heads). Each sub-model is specialized to predict the steering action for a specific future time step ($t + \delta_i$).
   • Output: A vector of predictive actions: $a^{TAPM}_t = [a_{t+\delta_1}, a_{t+\delta_2}, ..., a_{t+\delta_N}]$.

Mitigation Mechanism

The system operates as a time-offset control problem:

1. Latency Measurement: The real-time perception latency ($\delta_t$) is measured or estimated at runtime.
2. Interpolation: The final control action ($a^{PLM}_t$) is determined by linearly interpolating between the BM's current action (representing $\delta=0$) and the TAPM's discrete future predictions based on the measured $\delta_t$.
   • If $\delta_t$ matches a predefined latency head, that specific prediction is used.
   • If $\delta_t$ falls between heads, linear interpolation is applied to generate a smooth, continuous action.
3. Training Strategy:
   • BM Training: Standard supervised learning on expert driving data.
   • TAPM Training: A new dataset is generated where inputs are mapped to future ground-truth actions ($a_{t+\delta}$). The BM is frozen during this phase to ensure the TAPM learns to correct the latency without altering the base policy.

3. Key Contributions

1. Formulation: Framed perception latency in vision-based imitation learning as a time-offset control problem, analyzing how delayed observations degrade stability.
2. Modular Framework: Introduced PLM-Net, a plug-in architecture that augments existing controllers without retraining or modifying the base policy, preserving deployment characteristics.
3. Timed Action Prediction Model (TAPM): Developed a latency-conditioned multi-head predictive module that generates discrete future actions indexed by delay values, enabling adaptation to both constant and time-varying latency via runtime interpolation.
4. Validation: Demonstrated significant performance improvements in a deterministic closed-loop simulation environment, isolating latency effects from speed variations.

4. Experimental Results

The framework was evaluated in the OSCAR simulator using a Ford Fusion model on a test track with straight segments and curves. The vehicle speed was held constant to isolate latency effects.

Performance Metrics

• Steering Angle Similarity: Measured via Mean Absolute Error (MAE), Mean Squared Error (MSE), and Root Mean Squared Error (RMSE) against a latency-free baseline.
• Trajectory Similarity: Measured via Partial Curve Mapping, Fréchet distance, Area between curves, and Driving Trajectory Stability Index (DTSI).

Key Findings

• Constant Latency (0.2s):
  • The baseline model (BM) with latency showed a significant degradation in steering accuracy.
  • PLM-Net reduced the MAE by 62.1% compared to the latency-affected BM.
  • Trajectory deviation (Partial Curve Mapping) increased by only 11.1% with PLM-Net, whereas the BM without mitigation saw a 238.7% increase.
• Time-Varying Latency (0.0s – 0.35s):
  • PLM-Net successfully adapted to fluctuating delays.
  • MAE reduction reached 78.7% compared to the latency-affected BM.
  • Trajectory stability metrics (Fréchet distance, DTSI) showed substantial improvements, confirming the system's ability to maintain lane centering under dynamic delay conditions.
• Computational Cost:
  • Inference time increased by approximately 3.56 ms per frame (from 2.35 ms to 5.91 ms).
  • GPU memory usage increased by ~128 MB (19.7%).
  • The system remains real-time feasible at 30 Hz control frequency with a substantial timing margin.

5. Significance and Conclusion

PLM-Net represents a significant advancement in robust autonomous driving control by addressing the inevitable reality of perception latency.

• Practicality: Unlike model-based approaches requiring complex system identification, PLM-Net is a data-driven solution that can be deployed as a "plug-in" to existing imitation learning pipelines.
• Robustness: It effectively handles both fixed and fluctuating delays, a common issue in real-world AV systems due to variable computational loads and network jitter.
• Future Work: The authors note limitations regarding the fixed-speed simulation environment and the bounded latency range (0–0.35s). Future research aims to extend the latency range, test at varying speeds, and integrate the framework into real-world hardware and higher-fidelity simulators.

In summary, PLM-Net provides a computationally efficient, modular, and highly effective method for compensating perception latency, significantly enhancing the safety and stability of vision-based lateral control systems.