Deconfounded Lifelong Learning for Autonomous Driving via Dynamic Knowledge Spaces

This paper proposes DeLL, a deconfounded lifelong learning framework for end-to-end autonomous driving that integrates a Dirichlet process mixture model with front-door causal adjustment to dynamically expand knowledge spaces, mitigate catastrophic forgetting, and eliminate spurious correlations for improved adaptability and performance.

The Gist

Imagine you are teaching a robot to drive a car. The goal is for this robot to learn how to handle everything: stopping for red lights, merging onto highways, parking in tight spots, and reacting to sudden emergencies.

The problem with current robot drivers is that they suffer from "Catastrophic Forgetting." It's like a student who studies for a math test, passes it, but then immediately forgets how to read when they start studying history. Every time the robot learns a new driving skill, it overwrites its old memories, causing it to crash or ignore traffic signs it used to know perfectly.

Furthermore, these robots often get confused by "Spurious Correlations." Imagine a robot learns that "whenever there is a red truck, I should stop." It doesn't realize the truck is just a truck; it thinks the color red is the reason to stop. If it sees a red fire hydrant, it might panic and stop. It's learning the wrong cause-and-effect relationships.

The paper introduces a new system called DeLL (Deconfounded Lifelong Learning) to fix these problems. Here is how it works, using simple analogies:

1. The Infinite Library (Dynamic Knowledge Spaces)

Most robots have a fixed-size brain. Once it's full, they have to delete old files to make room for new ones.

DeLL is different. It uses a special mathematical tool called a Dirichlet Process Mixture Model (DPMM). Think of this as an infinite, self-expanding library.

• The Feature Library: This part of the library stores "concepts" (like what a rainy road looks like vs. a sunny one).
• The Trajectory Library: This part stores "moves" (like how to merge, how to turn sharply, how to park).

As the robot drives and encounters new situations, this library automatically creates new shelves for new types of knowledge. It never runs out of space, and it never has to throw away old books. This solves the "forgetting" problem because the robot keeps everything it has ever learned, organized neatly on its own shelves.

2. The Detective (Causal Front-Door Adjustment)

Remember the robot that stopped for red trucks? It was confused by a "confounder" (the color red). It needed a detective to figure out the real reason to stop.

DeLL uses a technique called Front-Door Adjustment. Imagine the robot is trying to decide whether to brake.

• Old Way: It looks at the camera (Input) and immediately hits the brakes (Output). It might get tricked by noise or bad lighting.
• DeLL Way: It inserts a mediator (the Detective).
  1. The robot looks at the scene.
  2. It asks the Detective: "Based on my library of past experiences, what is the true driving pattern here?" (e.g., "Is this a pedestrian? Is this a stop sign?").
  3. The Detective filters out the noise (like a red truck that isn't a threat) and only passes the true causal signal to the brakes.

This ensures the robot learns the real rules of driving, not just coincidental patterns.

3. The Evolutionary Decoder (The Flexible Driver)

Traditional robots plan their path step-by-step, like reading a book one word at a time. If they get stuck, they have to start over.

DeLL uses an Evolutionary Trajectory Decoder. Think of this as a chess player who can look at the whole board at once.

• Instead of planning one move at a time, the robot looks at its entire library of "moves" (the Trajectory Library).
• It picks the best set of moves from its library simultaneously (in parallel) to create a smooth path.
• As the robot learns more, its library of moves grows, and it gets better at picking the right combination instantly.

The Result: A Super-Student Driver

The researchers tested this in a high-fidelity driving simulator (CARLA). They taught the robot a series of difficult tasks one by one (like emergency braking, then merging, then parking).

• The Old Robots: By the time they learned to park, they had forgotten how to brake for emergencies. They crashed.
• DeLL: It learned all the skills, kept all the old skills, and even got better at the old ones because it could transfer knowledge between them. It didn't just learn to drive; it learned to drive forever without losing its memory.

In short: DeLL gives the robot an infinite, organized memory and a "truth detector" to ensure it learns the right lessons, allowing it to become a driver that gets smarter every day without ever forgetting how to drive.

Technical Summary

1. Problem Statement

End-to-End Autonomous Driving (E2E-AD) systems face three critical challenges when deployed in open-world, non-stationary environments:

• Catastrophic Forgetting: Standard neural networks struggle to learn new driving scenarios without overwriting or degrading performance on previously learned tasks.
• Causal Confusion & Spurious Correlations: Models often rely on unobservable confounders (e.g., sensor noise, specific environmental textures) rather than true causal driving intents, leading to decision-making failures under covariate shifts.
• Rigidity of Fixed Architectures: Existing methods typically use fixed-capacity networks or rigid task boundaries, which cannot adaptively expand to accommodate an infinite stream of new driving abilities without predefining the number of clusters or relying on expensive experience replay buffers.

2. Methodology: The DeLL Framework

The authors propose DeLL, a framework that integrates Dirichlet Process Mixture Models (DPMM) with Front-Door Adjustment from causal inference. The architecture consists of four core components:

A. Multi-modal Perception Backbone

• Utilizes Transfuser++ as the base extractor to process RGB sequences and LiDAR point clouds.
• Employs cross-modal attention to generate high-dimensional Bird's Eye View (BEV) feature maps.
• Includes auxiliary tasks (semantic segmentation and detection) to enrich geometric and semantic boundaries.

B. Dynamic Dual Knowledge Spaces (Based on DPMM)

To overcome memory bottlenecks and enable infinite learning, DeLL constructs two dynamic knowledge spaces using DPMM, a non-parametric Bayesian model that determines the number of clusters automatically:

1. Feature Knowledge Space (FKS): An implicit space that clusters fused BEV features to discover latent topological causal structures in the environment. It generates Feature Anchors.
2. Trajectory Knowledge Space (TKS): An explicit kinematic space that clusters ground-truth expert trajectories (e.g., lane changes, sharp turns) to build a physical prior action library. It generates Trajectory Anchors.

• Mechanism: The DPMM allows for the dynamic instantiation of new cluster components (anchors) as new scenarios are encountered, preserving historical knowledge in isolated, specialized distributions without rigid task boundaries.

C. Causal Feature Enhancement Module

This module addresses spurious correlations using the Front-Door Adjustment mechanism ($X \to M \to Y$), where the DPMM-derived knowledge anchors serve as the mediator variable $M$.

• Fused Feature Enhancement Module (FFEM): Processes raw multi-modal features. It uses cross-attention to map input features to the Feature Anchors, effectively calculating the expectation $P(m|x)$ to find the most fitting historical causal template.
• Trajectory Feature Enhancement Module (TFEM): Applies the same causal intervention to trajectory features, imposing kinematic constraints derived from the Trajectory Anchors.
• Result: This "deconfounds" the features, filtering out noise and environmental biases to enhance causal expressiveness.

D. Evolutionary Trajectory Decoder

• Replaces traditional autoregressive decoders with a non-autoregressive, parallel generation strategy.
• Dynamically expands the "token pool" of trajectory anchors as the DPMM learns new clusters.
• Uses a dual-branch head: a coarse-grained branch to select the best anchor (Top-K routing) and a fine-grained branch to predict geometric offsets.

3. Key Contributions

1. DeLL Framework: A novel deconfounded lifelong learning architecture for E2E-AD that integrates causal inference with dynamic knowledge spaces.
2. DPMM-based Dual Knowledge Spaces: A mechanism to dynamically preserve and update both latent feature representations and explicit trajectory priors without predefining cluster counts.
3. Causal Feature Enhancement: A novel application of front-door adjustment using knowledge anchors as mediators to mitigate spurious correlations caused by unobservable confounders.
4. Evolutionary Trajectory Decoder: A decoder that natively supports parallel trajectory generation and adapts to the growing knowledge base.
5. New Evaluation Protocol: Introduction of a lifelong learning benchmark based on Bench2Drive, featuring specific metrics for vertical stability (forgetting) and horizontal transferability.

4. Experimental Results

The method was evaluated in the CARLA simulator using the Bench2Drive benchmark, which involves five sequential tasks (Emergency Braking, Traffic Sign Recognition, Merging, Overtaking, Giving Way) with decreasing data volume.

• Lifelong Learning Performance:
  • Forgetting: Significantly reduced the Forgetting Ratio (FR) from 44.50% (baseline) to 33.97%.
  • Transfer: Improved Backward Transfer (BT) from 52.83% to 79.63%, indicating that learning new tasks actually reinforced previous knowledge.
  • Overall Score: Achieved an average Driving Score of 74.69% and Success Rate of 50.73%, outperforming the baseline TF++ significantly, especially in data-scarce scenarios like "Giving Way."
• Full-Data Performance:
  • Achieved State-of-the-Art (SOTA) performance among all compared E2E-AD models, with a Driving Score of 86.86% and Multi-Ability Success Rate of 68.9%.
• Ablation Studies: Confirmed that removing any core component (Evolutionary Decoder, TFEM, or FFEM) led to performance degradation, proving the necessity of the synergistic design.
• Visualization: Qualitative results showed that DeLL retained speed control strategies and lane-changing skills after learning new tasks, whereas the baseline forgot these behaviors or learned incorrect causal relationships (e.g., associating stationary vehicles with zero velocity regardless of context).

5. Significance

• Paradigm Shift: Moves E2E-AD from static, finite training to a dynamic, lifelong learning paradigm that mimics human adaptability.
• Causal Robustness: By explicitly addressing unobservable confounders via front-door adjustment, the system becomes more robust to environmental shifts and sensor noise, a critical requirement for real-world deployment.
• Scalability: The non-parametric nature of DPMM allows the system to scale indefinitely to new driving scenarios without architectural re-engineering or massive memory buffers.
• Benchmarking: Establishes a rigorous evaluation standard for lifelong learning in autonomous driving, filling a gap in the CARLA ecosystem.

Limitations: The alternating optimization between DPMM and network training introduces computational overhead, and the current validation is limited to simulation (CARLA), leaving the real-world domain gap as an open challenge.