The Role of Road Features and Vehicle Dynamics in Cost-Effective Autonomous Vehicles Safety Testing: Insights from Instance Space Analysis

1. Problem Statement

Autonomous Vehicle (AV) safety testing faces a critical bottleneck: real-world testing is prohibitively expensive and time-consuming, while simulation-based testing, though cost-effective, suffers from an immense search space.

• The Challenge: Generating safety-critical test cases (scenarios that expose AV failures) is difficult due to the complexity of dynamic environments and the sheer number of possible test configurations.
• The Gap: Existing research typically analyzes static features (road geometry, infrastructure) and dynamic features (vehicle behavior, sensor data) in isolation. There is a lack of systematic understanding regarding how these two feature types interact and synergistically influence test outcomes (i.e., whether a scenario leads to a safety violation).
• The Objective: To develop a unified framework that analyzes the inter-dependencies between static and dynamic features to improve test case generation, fault detection, and outcome prediction without executing every simulation.

2. Methodology

The study employs an integrated approach centered on Instance Space Analysis (ISA) and Machine Learning (ML) to investigate the relationship between road characteristics and AV behavior.

A. Data Source

• Dataset: The study utilizes SensoDat, a large-scale dataset containing over 32,000 test cases generated by three search-based techniques (Frenetic, FreneticV, AmbieGen) within the BeamNG.tech simulator.
• Selection: A balanced subset of 6,122 test cases was selected, ensuring an equal proportion of "effective" (failure-inducing) and "ineffective" (safe) cases.
• Feature Extraction:
  • Static Features (19 total): Derived from road coordinates (e.g., curvature, lane structure, angles, road length) using the SDC-Scissor tool.
  • Dynamic Features (181 total): Derived from AV sensor data and internal states (e.g., steering angle, acceleration, altitude, Electronic Stability Control activation) using descriptive analytics.

B. Instance Space Analysis (ISA)

ISA is used to map high-dimensional test cases into a 2D space to visualize the relationship between features and outcomes.

1. Feature Selection: An iterative process using Pearson correlation and K-means clustering identifies the most impactful features that distinguish effective from ineffective cases.
2. Projection: The PILOT (Projecting Instances with Linearly Observable Trends) algorithm projects the selected features into a 2D instance space. This ensures that:
   • Test cases with similar outcomes cluster together.
   • Linear trends between feature values and outcomes are preserved.
3. Visualization: The resulting 2D map allows researchers to visually inspect how specific ranges of static features (e.g., sharp curves) correlate with dynamic responses (e.g., steering adjustments) to cause failures.

C. Predictive Modeling (Regression Testing)

To answer whether test outcomes can be predicted before execution:

• Models: Five ML classifiers were trained: Random Forest (RF), Decision Tree (DT), K-Nearest Neighbors (KNN), Multilayer Perceptron (MLP), and Naive Bayes (NB).
• Configurations: Models were trained under three conditions:
  1. Static features only.
  2. Dynamic features only.
  3. Combined static and dynamic features.
• Evaluation: Performance was measured using Precision, Recall, and F1-Score on a held-out 20% test set.

3. Key Contributions

1. Unified Framework: A novel approach that concurrently analyzes static (road) and dynamic (vehicle) features, bridging the gap between infrastructure-based and behavior-based testing.
2. Identification of Inter-dependencies: The study reveals that static road features (e.g., curvature angles) directly dictate dynamic vehicle responses (e.g., steering variability and stability control activation), creating a synergistic effect on safety outcomes.
3. Predictive Capability: Demonstrated that ML models trained on the combined feature set significantly outperform those trained on single feature types, enabling the prediction of safety-critical test cases without simulation execution.
4. Visual Insight: Provided a 2D instance space visualization that helps practitioners understand why certain road geometries lead to AV failures.

4. Key Results

A. Most Impactful Features (RQ1)

ISA identified 10 critical features (5 static, 5 dynamic) that best predict test outcomes:

• Static: full_road_diversity (curve variety), max_angle, min_angle, std_angle (road angle variability), and road_distance.
• Dynamic: steering_std (steering variability), steering_min, steering_input_max, altitude_max (vehicle tilt), and esc_std (stability control activation frequency).

B. Interaction of Features (RQ2)

• Mechanism: Effective (failure) test cases typically occur on roads with long straight segments followed by sharp curves.
• Dynamic Response: This geometry forces the AV to maintain high speed on straights, then execute abrupt steering adjustments. This leads to high steering_std, significant altitude_max (due to lateral forces/tilting), and frequent ESC activation.
• Conclusion: Static features act as the "trigger," while dynamic features represent the "reaction." Analyzing them separately misses this causal chain.

C. Prediction Accuracy (RQ3)

The study compared ML performance across feature configurations. The Random Forest (RF) model achieved the best results:

• Combined Features (Static + Dynamic): Precision 0.958, Recall 0.915, F1-Score 0.936.
• Static Only: F1-Score ~0.889.
• Dynamic Only: F1-Score ~0.783.
• Finding: Integrating both feature types yields a statistically significant improvement in predictive accuracy, confirming that road geometry and vehicle dynamics are inseparable in safety assessment.

5. Significance and Implications

• Cost Reduction: By predicting test outcomes without running simulations, practitioners can prioritize the most likely failure-inducing scenarios, drastically reducing computational costs and testing time.
• Targeted Fault Detection: The approach allows for the identification of specific AV components (e.g., steering actuators, stability control) that fail under specific road conditions, enabling more targeted debugging.
• Optimized Regression Testing: In regression testing, where re-running all tests is expensive, this method allows for the selection of a minimal, high-impact subset of test cases that maximizes fault detection.
• Generalizability: While the study focused on Out-of-Bounds (OOB) incidents, the methodology is adaptable to other safety metrics (e.g., collision probability, time-to-collision) and different AV domains.

In summary, this paper establishes that holistic testing—which considers the interplay between the road environment and vehicle dynamics—is essential for effective AV safety validation. The proposed ISA-based framework provides a data-driven path to more efficient, accurate, and cost-effective autonomous system testing.