An End-to-End Decision-Aware Multi-Scale Attention-Based Model for Explainable Autonomous Driving

1. Problem Statement

Deep Neural Networks (DNNs) are critical for autonomous driving but operate as "black boxes," lacking transparency in their decision-making processes. This opacity hinders trust, system troubleshooting, and regulatory acceptance. Existing Explainable AI (XAI) methods for autonomous driving suffer from two fundamental flaws:

1. Decision Unawareness: Previous reasoning heads generate explanations without explicitly knowing the specific driving decision the network made, leading to logical inconsistencies.
2. Accuracy vs. Logic: High prediction accuracy does not guarantee logical reasoning. A model might predict the correct action for the wrong reason, which is dangerous in safety-critical applications.

The core challenge is to develop an end-to-end system that not only predicts driving actions (e.g., stop, turn) but also generates logically consistent textual and visual explanations that are aware of the specific decision made.

2. Methodology

The authors propose a novel End-to-End Decision-Aware Multi-Scale Attention-Based Model. The architecture integrates semantic segmentation, multi-scale feature extraction, and a hierarchical reasoning mechanism.

A. Architecture Overview

• Backbone & Feature Extraction: The model uses ResNet50 pre-trained on ImageNet and DeepLabV3 pre-trained on the BDD-10K dataset for semantic segmentation.
• Multi-Scale Atrous Spatial Pyramid Pooling (MASPP): To handle objects of varying sizes in driving scenes, the model aggregates features from ResNet layers 2, 3, and 4. It applies atrous convolutions with varying rates (12, 24, 36) to capture context at different scales.
• DCA-CBAM-ResNet50 Attention Module: A hybrid attention mechanism combining CBAM (Convolutional Block Attention Module) and DCANet.
  • It employs Channel Attention and Spatial Attention sequentially.
  • Inter-block Connections: A unique feature where attention weights from previous blocks are fed into subsequent blocks (via a learnable connection function) to gradually refine features and prevent drastic shifts in attention characteristics.
• Multi-Task Heads:
  1. Action Head: Predicts the driving decision (e.g., Move forward, Stop, Turn left/right).
  2. Reasoning Head: Generates the explanation (e.g., "Obstacle on left," "Red light").
  • Decision-Aware Mechanism: The critical innovation is that the sigmoid probabilities of the Action Head are concatenated with the feature maps and fed into the Reasoning Head. This ensures the explanation module is explicitly aware of the predicted decision before generating reasons.

B. Loss Function

The model utilizes a multi-task loss function:
$$L = L_{action} + \lambda L_{reason}$$
Where $\lambda$ controls the relative importance of the reasoning task versus the action task.

C. Evaluation Metric: Joint F1 Score

The authors argue that standard F1 scores for action and reason separately are insufficient. They propose the Joint F1 Score, which evaluates whether the model makes the correct decision for the correct reason.

• It calculates the F1 score based on valid action-reason pairs (e.g., "Turn Right" is only valid with reasons like "Right lane clear," not "Obstacle on left").
• The metric is defined over 33 valid action-reason combinations derived from the dataset logic.

D. Visual Explanation

The model generates pixel-level visual explanations using Grad-CAM, weighted by the logits of both the action and reason outputs (equal weights of 0.5), to highlight the image regions influencing the decision.

3. Key Contributions

1. Novel Network Architecture: An end-to-end multi-task model with MASPP and a hybrid DCA-CBAM attention mechanism that captures both fine-grained details and global context.
2. Decision-Aware Reasoning: A mechanism where driving decisions are explicitly fed into the reasoning module, ensuring explanations are logically tied to the specific action taken.
3. Joint F1 Score: A new evaluation metric designed to measure the reliability of XAI by assessing the alignment between actions and their reasons, rather than treating them as independent tasks.
4. Comprehensive Validation: Extensive testing on two datasets (BDD-OIA and nu-AR) demonstrating superior performance over state-of-the-art (SOTA) models.

4. Experimental Results

The model was evaluated on the BDD-OIA (training/validation/testing) and nu-AR (generalization) datasets.

• Performance on BDD-OIA:
  • Explanation Accuracy: Achieved an Overall F1 Score of 0.551 and a Mean F1 Score of 0.384 for explanations, outperforming all SOTA models (e.g., OIA, NLE-DM, CBM-AUC).
  • Joint F1 Score: Achieved 48.73% (Overall) and 36.87% (Mean), significantly higher than the NLE-DM baseline (46.2% and 31.9%), proving the model makes correct decisions for correct reasons more frequently.
• Generalization (nu-AR):
  • The model outperformed OIA and NLE-DM on the nu-AR dataset, showing robustness across different environments and datasets.
• Ablation Studies:
  • $\lambda$ Parameter: Setting $\lambda=2$ (prioritizing reason learning) yielded the best action performance, while $\lambda=1$ offered the best balance. Multi-task learning improved action performance compared to an action-only network.
  • Reduction Ratio: A reduction ratio of 16 in the attention module provided the best trade-off between parameter efficiency and reasoning accuracy.
• Qualitative Analysis: Visualizations (Grad-CAM) showed the model correctly focusing on relevant objects (pedestrians, traffic lights) and successfully identifying cases where human annotations were incorrect, demonstrating the model's potential to correct noisy ground truth.

5. Significance and Conclusion

This research addresses the critical "black box" problem in autonomous driving by ensuring that explanations are not just post-hoc approximations but are logically derived from the decision process.

• Trust & Safety: By verifying that the "reason" matches the "action," the system increases trust for users and regulators.
• System Improvement: The ability to identify when a model makes a correct decision for the wrong reason (or vice versa) allows for targeted debugging and faster development cycles.
• Future Impact: The introduction of the Joint F1 score sets a new standard for evaluating XAI in safety-critical domains, moving beyond simple accuracy to assess the causal logic of AI decisions. The authors suggest future work should focus on relabeling datasets to treat (action, reason) as a single joint annotation to further improve evaluation rigor.