PRAM-R: A Perception-Reasoning-Action-Memory Framework with LLM-Guided Modality Routing for Adaptive Autonomous Driving

This paper introduces PRAM-R, a unified framework that leverages an LLM-guided router and hierarchical memory within an asynchronous dual-loop architecture to dynamically optimize sensor modality usage, significantly reducing computational costs and routing instability while maintaining high trajectory accuracy in autonomous driving.

The Gist

Imagine you are driving a car, but instead of just you, there's a team of experts inside the vehicle helping you navigate. This team includes a Camera (the eyes), a LiDAR (a laser scanner that sees 3D shapes), a Radar (which sees through fog and rain), and a Map (the brain's memory of the road).

In most self-driving cars today, all these experts are shouting at the computer 24/7, even when they aren't needed. It's like having a choir of 100 people singing in a small room; it's loud, exhausting, and often redundant. This wastes battery power and slows down the car's computer.

This paper introduces PRAM-R, a smarter way to run a self-driving car. Think of it as upgrading the car's "brain" to be a smart manager who knows exactly who to listen to, when to listen, and what to remember.

Here is how PRAM-R works, broken down into simple parts:

1. The Two-Loop System: The "Fast Reflex" vs. The "Slow Thinker"

PRAM-R uses a clever two-speed system, like a human brain:

• The Fast Loop (Reflexes): This is the car's immediate reaction. It happens super fast (like catching a ball before you even think about it). It handles steering and braking right now.
• The Slow Loop (Deliberation): This is the car's "thinking" mode. It runs a bit slower. It looks at the big picture, checks the weather, looks at the map, and asks: "Hey, do we really need the laser scanner right now? It's foggy, so maybe just the radar and camera are enough."

The Analogy: Imagine you are walking through a dark forest.

• The Fast Loop is your feet moving quickly to avoid tripping.
• The Slow Loop is your brain pausing for a second to say, "Okay, it's foggy, I can't see far. I should turn on my flashlight (Camera) and rely on my hearing (Radar), but I don't need to scan the whole forest with a laser (LiDAR) because it's too thick."

2. The "Smart Manager" (The LLM Router)

At the heart of this system is a Large Language Model (LLM). Think of this as a very experienced, wise tour guide sitting in the driver's seat.

• Instead of just crunching numbers, this guide reads the situation. It looks at the sensor data and says, "The camera is blurry because it's raining, so let's trust the radar more. The LiDAR is working great, so let's keep it on."
• It decides which sensors to turn ON and which to turn OFF (or dim down) to save energy. It's like a smart home system that turns off lights in empty rooms but keeps them on in the kitchen.

3. The "Memory" (The Notebook)

One of the biggest problems with AI is that it often forgets what happened five seconds ago. PRAM-R has a Hierarchical Memory, which is like a multi-level notebook:

• Short-term Memory: "I just saw a red light 2 seconds ago."
• Mid-term Memory: "I've been driving in the rain for 10 minutes; the road is slippery."
• Long-term Memory: "I've driven this specific intersection 50 times before; I know the traffic light is tricky here."

Why is this cool?
Without memory, the AI might panic every time the rain starts, thinking it's a new problem. With memory, it remembers, "Oh, I've handled rain before. I know to switch to Radar mode." This stops the car from constantly flipping sensors on and off (which is called "oscillation" in the paper) and keeps the ride smooth.

4. The Results: Smarter, Faster, and Cheaper

The researchers tested this system in two ways:

1. Simulated Stress Tests: They threw fake problems at the car (sudden sensor failures, heavy rain, flickering lights).
   • Result: The system was 87% more stable. It didn't panic and flip switches wildly; it stayed calm and stuck to the best sensors.
2. Real-World Test (nuScenes Dataset): They tested it on real driving data.
   • Result: The car used 6% fewer sensors on average (saving power and computing power) but drove just as safely and accurately as cars that use all sensors all the time.

The Big Picture

PRAM-R is about efficiency. It teaches the self-driving car to be a "smart consumer" of its own sensors. Instead of burning energy by using everything all the time, it uses its "brain" (the LLM) and its "memory" to pick the perfect tool for the job at hand.

It's the difference between a student who studies every single page of a textbook for every exam (wasting time and energy) versus a student who knows exactly which chapters to review based on the specific questions being asked (smart, fast, and effective).

Technical Summary

Here is a detailed technical summary of the paper "PRAM-R: A Perception-Reasoning-Action-Memory Framework with LLM-Guided Modality Routing for Adaptive Autonomous Driving."

1. Problem Statement

Autonomous driving systems rely on multimodal perception (cameras, LiDAR, radar, maps) to ensure robustness. However, continuously activating all sensors incurs significant computational overhead, latency, and energy consumption, especially when redundant or low-quality sensors are included.

• Limitations of Existing Approaches: Current methods often use static fusion (activating all sensors) or heuristic/attention-based routing. These lack high-level reasoning capabilities, struggle to generalize across diverse scenarios, and fail to balance perception accuracy with computational efficiency.
• LLM Challenges: While Large Language Models (LLMs) offer powerful reasoning, applying them directly to real-time driving is hindered by sequential inference latency (if invoked per frame) and the lack of memory, which prevents the reuse of past routing experiences, leading to redundant reasoning.

2. Methodology: The PRAM-R Framework

The authors propose PRAM-R, a unified Perception–Reasoning–Action–Memory framework designed for adaptive autonomous driving. It integrates an LLM (specifically Qwen3-VL-8B) as a modality router within a novel asynchronous dual-loop architecture.

A. Asynchronous Dual-Loop Architecture

To address latency and adaptability, the system decouples operations into two loops:

1. Fast Reactive Loop: Operates at high frequency for real-time perception and control. It uses the latest available memory state ($M_\tau$) to generate immediate driving commands ($O_t$).
2. Slow Deliberative Loop: Operates at a lower frequency to perform high-level reasoning, modality selection, and memory updates. It aggregates historical data to refine sensing strategies and update the memory state ($M_t$).

B. Core Modules

1. Perception Layer (with LLM-Guided Routing):

   • Shallow Perception: Extracts diagnostic indicators (e.g., brightness, point density, noise ratio) from sensors and environmental context (weather, map complexity).
   • Modality Routing: The LLM analyzes these indicators to output:
     • Usage Masks ($u_i$): Based on scene complexity (which sensors are sufficient).
     • Reliability Scores ($r_i$): Based on sensor health and diagnostics.
   • Selection Logic: The final active set is the intersection of "sufficient" and "reliable" sensors. If the intersection is empty, it falls back to the reliable set.
   • Stabilization: To prevent rapid switching (oscillations), the system employs hysteresis (different thresholds for turning ON vs. OFF) and Exponential Moving Average (EMA) smoothing on weights.

2. Reasoning and Action Layers:

   • Follows the design of the PLA framework [8], using the LLM to transform fused perception into semantic representations, assess risks, and generate trajectory plans. These layers are coupled with the memory module for context-aware decision-making.

3. Hierarchical Memory Architecture:

   • Designed to enable long-term adaptation and reduce inference overhead by caching past decisions. It consists of four layers operating at different temporal scales:
     • Perception Memory: Short-term (P-Routing Context, P-Semantic Cache) for immediate diagnostic tracking.
     • Reasoning Memory: Mid-term (R-Scene Record, R-Seed State) for maintaining decision consistency and interpretability.
     • Action Memory: Mid-term (A-Policy Log) for tracking control commands and errors.
     • Knowledge Base: Long-term experiential memory for distilling optimal strategies and scene descriptors for continual self-improvement.

3. Key Contributions

1. PRAM-R Framework: A novel integration of LLM-guided modality routing into the autonomous driving pipeline, enabling adaptive and efficient multimodal perception.
2. Reasoning-Driven Routing: A mechanism that dynamically selects and weights sensors by integrating sensor diagnostics with environmental context, optimizing the trade-off between efficiency and reliability.
3. Asynchronous Dual-Loop Design: Decouples fast reactive control from slow deliberative reasoning, achieving low-latency operation while maintaining robustness in dynamic scenarios.
4. Hierarchical Memory Module: A multi-scale memory system that caches routing decisions, enforces temporal consistency, tracks sensor reliability patterns, and preserves contextual knowledge for continual adaptation.

4. Experimental Results

The framework was evaluated using synthetic stress tests and real-world data from the nuScenes dataset.

• Routing Stability (Synthetic Stress Tests):

  • Under high-frequency perturbations (simulating noise and rapid changes), the hysteresis mechanism reduced modality switching (oscillations) by 87.2% compared to a simple threshold-based approach.
  • This confirms the system's ability to maintain stable sensor activation without jitter.

• Real-World Validation (nuScenes):

  • Efficiency: Achieved a 6.22% reduction in active modalities (sensing load) while maintaining a 20% memory recall rate.
  • Performance: Trajectory accuracy (measured by Average Displacement Error - ADE and Final Displacement Error - FDE) remained comparable to full-modality baselines (Static Fusion/PLA).
  • Adaptivity: In medium-complexity scenes, the system successfully deactivated LiDAR (relying on Camera and Radar), while engaging all sensors in high-complexity scenarios.

• Ablation Study:

  • Removing the memory or the dual-loop design resulted in lower routing efficiency and stability (RSI), confirming that both components are essential for optimal performance.
  • The full PRAM-R system achieved the highest Routing Efficiency (RE) and Stability Index (RSI) with negligible loss in routing consistency (RC).

5. Significance and Future Work

• Significance: PRAM-R demonstrates that LLM-augmented architectures, when paired with hierarchical memory and asynchronous loops, can achieve efficient, adaptive multimodal perception. It solves the "latency vs. reasoning" dilemma by offloading complex routing decisions to a slower loop while maintaining real-time control.
• Future Directions:
  • Memory Validation: Conducting layer-wise ablations to quantify the specific impact of each memory layer.
  • Real-World Deployment: Expanding evaluation to real-vehicle logs and digital twins to bridge the simulation-to-reality gap.
  • Optimization: Investigating model size and inference latency to better balance reasoning capabilities with the constraints of embedded hardware.

In conclusion, PRAM-R represents a significant step toward energy-efficient and robust autonomous driving by intelligently managing sensor resources through reasoning and memory, rather than brute-force data processing.