Beyond Fixed Thresholds and Domain-Specific Benchmarks for Explainable Multi-Task Classification in Autonomous Vehicles

This paper proposes an adaptive threshold selection methodology to overcome the limitations of fixed thresholds in multi-task autonomous driving perception and introduces the IUST-XAI-AD dataset to enable cross-cultural explainability and more robust evaluation of autonomous vehicle systems.

The Gist

Imagine you are teaching a robot to drive a car. You want the robot not just to know what to do (like "stop" or "turn left"), but also to explain why it's doing it (like "because there's a pedestrian" or "because the light is red"). This is the goal of Explainable AI in self-driving cars.

However, there are two big problems the authors of this paper found:

1. The "One-Size-Fits-All" Rule: Most robots are programmed with a rigid rule: "If you are more than 50% sure, make a decision." The authors call this a "fixed threshold." They argue this is like telling a human, "If you are 50% sure it's raining, grab an umbrella." That doesn't work well! Sometimes you need to be 90% sure before you act (like stopping for a child), and sometimes 50% is fine. The paper shows that using a single 50% rule for every situation makes the robot make more mistakes.
2. The "Western Bias" in Training: Most robots are trained on data from places like California or Germany. But driving in Tehran, Iran, is very different. There are more motorcycles, different traffic habits, and different road layouts. If you only train a robot on Western roads, it might get confused when it sees a chaotic Middle Eastern street.

Here is how the authors fixed these problems, explained simply:

1. Tuning the "Confidence Dial" (The Threshold)

Think of the robot's brain as having a volume dial for every decision.

• The Old Way: Everyone turned the dial to exactly "5" (50% confidence) and never touched it again.
• The New Way: The authors tested the dial at every setting from 1 to 10. They found that for some tasks (like deciding to "stop"), the robot works best when the dial is set to "3" (30% confidence). For other tasks (like explaining why it stopped), "4" (40%) is better.

The Analogy: Imagine you are a security guard checking IDs.

• If you are too strict (high threshold), you let no one in, even if they are friendly (you miss good opportunities).
• If you are too loose (low threshold), you let in everyone, including bad actors (you make dangerous mistakes).
• The authors found that for different types of "bad actors" (different driving tasks), you need a different level of strictness. By adjusting the "strictness dial" for each specific job, the robot became much smarter and safer.

2. The New "Middle Eastern Driving School" (The Dataset)

The authors realized that existing driving datasets were like a driving school that only taught you how to drive on empty, straight highways in Europe. They didn't teach you how to handle a busy, chaotic market street in Iran.

• The Solution: They created a new dataset called IUST-XAI-AD.
• What's in it: 958 real photos taken in Qom, Iran.
• Why it's special: It's like a "hard mode" level in a video game. It has way more motorcycles, more pedestrians, and more complex traffic patterns than the standard datasets.
• The Result: When they tested their robot on this new "hard mode," it struggled more than on the easy European roads. This proves that the new dataset is a better, tougher test to see if a robot is truly ready for the real world.

3. The "Why" Matters Just as Much as the "What"

The robot has to do two things at once:

1. Action: "Stop the car."
2. Reason: "Because a person is crossing."

The authors found that the robot is actually better at guessing the action (Stop/Go) than it is at guessing the reason (Why?). It's like a student who can answer "True/False" questions easily but struggles to write the essay explaining why the answer is true. By using their new "tuned dials" (thresholds), they helped the robot get better at both the action and the explanation.

The Bottom Line

The paper says:

• Stop using the same 50% rule for everything. Adjust your confidence levels based on the specific task.
• Don't just test robots on Western roads. You need to test them on diverse, chaotic roads (like those in the Middle East) to see if they are truly safe.
• Explainability is key. A self-driving car isn't just a machine; it needs to be able to tell you why it made a decision so humans can trust it.

By fixing the "dials" and testing on "tougher roads," the authors have built a better foundation for self-driving cars that can be trusted anywhere in the world, not just in places that look like California.

Technical Summary

Technical Summary: Beyond Fixed Thresholds and Domain-Specific Benchmarks for Explainable Multi-Task Classification in Autonomous Vehicles

Problem Statement
Autonomous driving systems rely on deep learning models that are often "black boxes," lacking the transparency required for safety-critical deployment and human trust. While multi-task learning frameworks have emerged to simultaneously predict driving actions and their underlying reasoning (Explainable AI or XAI), current approaches face two primary limitations. First, they predominantly rely on fixed confidence thresholds (typically 0.5) to convert continuous model outputs into discrete predictions. This "one-size-fits-all" approach ignores the varying complexity and class imbalances inherent in different prediction tasks, potentially leading to suboptimal performance or safety risks. Second, existing evaluation benchmarks, such as Berkeley Deep Drive (BDD-OIA) and nuScenes Action and Reasons (nu-AR), are heavily skewed toward Western driving contexts. This lack of cultural diversity limits the assessment of model generalizability across different traffic patterns, road infrastructures, and driving behaviors found globally, particularly in Middle Eastern regions.

Methodology
The authors propose a two-pronged methodological advancement to address these gaps:

1. Confidence Threshold Sensitivity Analysis:
   Instead of using a static threshold, the authors introduce a systematic framework to optimize decision boundaries for multi-task classification. They define separate confidence thresholds ($\tau_A$ for action and $\tau_R$ for reason) and perform a grid search across the range [0, 1] with a step size of 0.1. The framework evaluates four distinct metrics: F1-action-overall, F1-action-mean, F1-reason-overall, and F1-reason-mean. This allows for the identification of optimal threshold pairs that maximize performance for specific tasks, acknowledging that action and reason predictions may require different confidence levels.

2. IUST-XAI-AD Dataset Construction:
   To address the lack of culturally diverse benchmarks, the authors introduce the IUST-XAI-AD dataset. Collected in Qom, Iran, this dataset comprises 958 dash-cam images captured under various lighting conditions. Each image is manually annotated by experts with:

   • Action Labels: 4 categories (Move forward, Stop/slow down, Turn left, Turn right).
   • Reason Labels: 21 categories explaining the driving decision (e.g., "Obstacle: person," "Traffic light is green").
     The dataset is analyzed for complexity using a weighted score ($C$) that accounts for the density of pedestrians, riders, and vehicles, assigning higher weights to vulnerable road users (pedestrians and riders) to reflect their higher risk profile.

Key Contributions
The paper presents three interconnected contributions:

• Systematic Threshold Optimization: A demonstration that fixed thresholds are suboptimal for multi-task scenarios. The authors provide a methodology to identify task-specific optimal thresholds, revealing that performance peaks vary significantly between action and reason tasks.
• Novel Benchmark (IUST-XAI-AD): The introduction of a dataset specifically designed for explainable computer vision in Persian driving contexts. This dataset addresses the cultural and regional bias in existing benchmarks, offering a more challenging environment with higher object density and distinct traffic patterns.
• Comprehensive Cross-Context Validation: An extensive evaluation of a previously proposed attention-based multi-task model across three datasets (BDD-OIA, nu-AR, and IUST-XAI-AD). This includes a comparative analysis of model performance, complexity metrics, and feature embeddings to assess robustness across different cultural environments.

Experimental Results

• Threshold Sensitivity: Analysis on the BDD-OIA dataset reveals that the conventional 0.5 threshold is not optimal. The peak F1-action-overall score occurs at a threshold of 0.3 (71.85%), while the peak F1-reason-overall score occurs at 0.4 (54.77%). The study identifies a "robust operating region" between 0.3 and 0.5 where all metrics remain within 1% of their peak values, suggesting that while tuning is beneficial, a range of thresholds can yield near-optimal results.
• Dataset Complexity: The IUST-XAI-AD dataset exhibits significantly higher complexity (score of 2.0038) compared to BDD-OIA (0.8062) and nu-AR (0.5752). This is driven by a higher density of vehicles (1.66 per image vs. 0.70 in BDD-OIA) and a drastically higher density of riders (0.164 per image, which is 19–24 times higher than in Western datasets), reflecting mixed traffic conditions common in Middle Eastern cities.
• Model Performance: When tested on IUST-XAI-AD, the authors' attention-based model outperformed the NLE-DM baseline in 9 out of 12 metrics. However, overall performance metrics (F1 scores) dropped significantly on IUST-XAI-AD compared to BDD-OIA, indicating that the cultural and environmental differences present fundamentally new challenges that current models struggle to generalize to without specific adaptation.
• Feature Analysis: t-SNE visualizations of learned features show that the model successfully clusters data by both action classes and environmental context (e.g., separating traffic infrastructure from dynamic obstacles), though some overlap remains.

Significance and Claims
The paper claims that its work advances the state-of-the-art in explainable multi-task learning for autonomous driving by providing both methodological tools and practical evaluation resources.

• Methodological: It challenges the standard practice of fixed thresholding, arguing that threshold selection should be a design choice based on specific operational priorities (e.g., balancing precision and recall) rather than a fixed optimization problem.
• Practical: The IUST-XAI-AD dataset serves as a critical tool for evaluating the cross-cultural generalization of autonomous systems, highlighting systematic gaps in model robustness that single-domain evaluations miss.
• Broader Impact: The authors assert that their combined contributions accelerate the development of more reliable, explainable, and culturally adaptive autonomous driving systems suitable for global deployment. They emphasize that comprehensive validation across diverse contexts is essential for assessing the true safety and robustness of these systems.

The paper concludes by noting that while the current work establishes a baseline for cultural diversity and threshold optimization, future research should investigate fine-tuning and threshold sensitivity specifically for adverse weather conditions and further expand evaluation to a wider range of geographically diverse datasets.