Technical Summary: Adaptive Control in Autonomous Driving via Real-Time Recurrent RL

Problem Statement

Learning-based control policies for autonomous driving often fail when deployed in environments that differ from their training data. These "distribution shifts"—caused by subtle changes in system dynamics, sensor characteristics, or environmental conditions—can lead to catastrophic performance degradation. Traditional offline learning paradigms, where policies are trained once and deployed without adaptation, are insufficient for these non-stationary real-world settings. While online adaptation is required, existing methods often rely on meta-learning (requiring specific task data), test-time adaptation (relying on hand-crafted rewards), or conventional online RL that introduces latency by waiting for full episodes or mini-batches before updating. Furthermore, standard online learning in recurrent networks often requires Backpropagation Through Time (BPTT), which is memory-intensive and biologically implausible.

Methodology

The authors propose a hybrid learning pipeline that combines offline behavioral cloning with online Real-Time Recurrent Reinforcement Learning (RTRRL). This approach enables policies to adapt continuously at every time step without retraining from scratch or accessing large offline datasets.

Core Algorithm: RTRRL

RTRRL is a memory-efficient, biologically plausible algorithm that updates policy parameters at every time step using a forward-only pass, eliminating the need for BPTT.

• Gradient Computation: Instead of BPTT, the framework utilizes Real-Time Recurrent Learning (RTRL) or Random Feedback Local Online (RFLO) learning. These methods compute gradients online using eligibility traces and Jacobian approximations. RFLO, in particular, enforces locality and biological plausibility by using fixed random feedback matrices, reducing complexity from $O(n^3)$ to $O(n^2)$.
• Learning Objective: The algorithm combines temporal-difference (TD) learning with eligibility traces (TD($\lambda$)) to handle delayed rewards. It updates actor and critic networks based on the TD-error $\delta_t = r_t + \gamma\hat{v}(h_{t+1}) - \hat{v}(h_t)$.
• Architecture: The authors employ separate Recurrent Neural Networks (RNNs) for the policy (actor) and value function (critic), rather than a shared backbone. The actor receives encoded observations and outputs an action distribution, while the critic estimates the state value.

Model Integration: LrcSSM

A key methodological extension in this work is the integration of Liquid-Resistance Liquid-Capacitance State-Space Models (LrcSSMs) into the RTRRL framework.

• LrcSSM Characteristics: Unlike Linear Recurrent Units (LRUs) which use purely linear dynamics, or standard Continuous-Time RNNs (CT-RNNs) with fixed time constants, LrcSSMs introduce input- and state-dependent nonlinear time constants. This allows neurons to adapt their temporal behavior dynamically.
• Diagonal Structure: LrcSSMs retain a diagonal structure, which allows for the exact computation of RTRL gradients without the performance loss typically associated with approximations, making them computationally efficient for online learning.

Training Pipeline

1. Offline Pre-training: A policy is first trained via behavioral cloning on a dataset of human demonstrations. This involves a convolutional encoder (autoencoder for image reconstruction) and a recurrent policy network.
2. Online Fine-tuning: The pre-trained policy is deployed and fine-tuned in real-time using RTRRL. To prevent overfitting during single-environment fine-tuning (effective batch size of 1), a parameter change penalty ($L_\theta = \beta ||\theta_{pre} - \theta_t||^2$) is added to the loss function.

Experimental Setup

The approach was validated in two distinct settings:

1. Simulation (CarRacing): A modified OpenAI Gym environment where agents must follow a track centerline. The environment includes a penalty for wheel deviation.
2. Real-World (RoboRacer): A 1:10-scale autonomous vehicle equipped with a Dynamic Vision Sensor (DVS) (event camera). The task is line-following. This setup is notable for using event-based vision (capturing intensity changes rather than frames) on standard non-spiking hardware.

Key Results

Simulation (CarRacing)

• Performance: All model types (CT-RNN, LRU, LrcSSM) showed significant improvement after online fine-tuning compared to their pre-trained baselines.
• Superiority of LrcSSM: LrcSSM-based policies demonstrated the fastest and most consistent improvement across different random seeds and tracks.
• Adaptation: Fine-tuned policies successfully adjusted to distribution shifts, such as a simulated steering angle offset, recovering performance levels comparable to initial rewards.

Real-World Deployment (LineTracking)

• Event Camera Success: This work presents the first demonstration of online RL fine-tuning using event-camera observations on standard hardware in a closed-loop control setting.
• Model Failure and Success:
  • LRU Failure: Pre-trained LRU models failed to learn an effective line-following policy, unable to complete a single lap without intervention. The authors hypothesize that the purely linear recurrence of LRUs is insufficient to capture the discontinuous, threshold-like transitions inherent in event-frame data (switching between line-visible and line-absent states).
  • LrcSSM Success: In contrast, LrcSSMs successfully learned the task. The authors attribute this to the nonlinear state dynamics of LrcSSMs, which are essential for handling the specific characteristics of event-based control, while retaining the diagonal structure needed for efficient gradient computation.
• Robustness: The system adapted to real-world distribution shifts, including a 0.1 rad steering angle shift and visual disturbances caused by ambient sunlight. The reward initially degraded upon the shift but recovered to pre-shift levels within a few laps.

Significance and Claims

The paper claims several firsts and significant contributions to the field of autonomous control:

1. First Application of LrcSSM in Online RL: This is the first application of LrcSSMs within an online reinforcement learning setting, demonstrating their viability as a drop-in replacement for LRUs that offers superior expressivity.
2. Hybrid Learning Pipeline: The work proposes the first pipeline combining offline behavioral cloning with online real-time RTRRL fine-tuning, enabling continuous adaptation to non-stationary environments without full retraining.
3. Feasibility on Embedded Systems: The results validate that biologically plausible, real-time online learning algorithms are feasible on embedded robotic systems (RoboRacer) without requiring specialized neuromorphic chips.
4. Event-Based Online Control: It presents the first high-frequency, non-spiking machine learning application of online fine-tuning using event camera observations in a closed-loop control setting.
5. Synergy of Bio-Inspiration: The paper showcases the synergy between bio-inspired sensors (event cameras), bio-inspired recurrent models (LrcSSMs), and biologically plausible learning rules (RTRRL/RFLO).

The authors conclude that while learning-based control is vulnerable to distribution shifts, the proposed RTRRL-based approach allows deployed policies to maintain performance by continuously adapting in real-time, with LrcSSMs emerging as the most effective architecture for this purpose.