From Research to Practice: An Interactive Rapid Review of Autonomous Driving System Testing in Industry

1. Problem Statement

Autonomous Driving Systems (ADS) are increasingly deployed in real-world traffic, yet their testing remains a fundamental challenge due to open operational environments, complex scenarios, and a lack of established processes and metrics. A significant gap exists between academic research and industrial practice:

• The Gap: While extensive research exists on ADS testing, it is rarely validated in industrial settings. The applicability of research results to real-world constraints (e.g., toolchain integration, data availability, safety regulations) is often overlooked.
• Specific Focus: The industry is transitioning toward End-to-End (E2E) ADS architectures, which pose unique testing challenges compared to modular approaches. There is a lack of practitioner-driven reviews that systematically evaluate how academic interventions address these specific industrial needs.
• Core Question: How can academic research on ADS testing be effectively translated into industrial practice, particularly regarding E2E systems?

2. Methodology: Interactive Rapid Review

The authors employed an Interactive Rapid Review approach, collaborating with 21 practitioners from a leading automotive company (Volvo Cars). This method differs from traditional systematic literature reviews by involving practitioners throughout the entire research lifecycle to ensure relevance and context-awareness.

The process followed five key steps:

1. Preparation: A review team of researchers and practitioners established a collaboration protocol.
2. Identifying Review Questions:
   • Four practitioners proposed 12 initial challenges/questions regarding ADS testing.
   • Five practitioners (including some new to the initial list to mitigate bias) prioritized these via a voting process.
   • Result: Two critical questions were selected for deep analysis:
     • Q7: How does the transition to E2E architecture affect verification and validation strategies?
     • Q8: How can one argue for the completeness of testing for E2E-enabled ADS?
3. Searching and Selecting Papers:
   • Keywords were derived from Q7 and Q8 and searched across IEEE Xplore, ACM Digital Library, Scopus, and arXiv (2016–2026).
   • Initial Pool: 1,389 unique studies.
   • Selection: 17 papers were selected based on relevance to the specific review questions and explicit focus on E2E ADS testing.
4. Extracting and Synthesizing Data:
   • The team extracted 17 "Technological Rules" (structured representations of research contributions capturing intervention, context, and expected effect) from the selected papers.
   • These rules were presented to the 21 practitioners in a follow-up workshop to assess relevance, applicability, and adoption constraints.
5. Dissemination: Results were synthesized into a report and planned for internal seminars to facilitate knowledge transfer.

3. Key Contributions

1. Practitioner-Driven Challenge Identification: The study identifies 12 practical challenges in ADS testing, prioritizing E2E testing approaches and testing completeness as the most urgent industrial needs.
2. Systematic Evaluation of Research: It provides the first review of ADS testing research specifically filtered through an industrial lens, analyzing 17 studies against real-world constraints.
3. Technological Rules Synthesis: The authors synthesized research interventions into 17 actionable technological rules, categorizing them by approach (e.g., constraint-based, RL-based, generative models).
4. Gap Analysis: The study highlights specific barriers to adoption, such as the complexity of multi-stage pipelines, lack of unified frameworks, and the "sim-to-real" gap in scenario realism.

4. Key Results and Findings

A. Analysis of Q7: Approaches for E2E ADS Testing

The 16 papers addressing Q7 primarily focus on generating critical scenarios to test E2E systems. They were categorized into four technical approaches:

• Constraint/Search-Based (8 papers): Methods like CoCoMagic and EpiTESTER use evolutionary algorithms and metamorphic testing to find safety-critical scenarios.
  • Practitioner Feedback: Highly relevant for regression testing and identifying long-tail scenarios. However, adoption is limited by the need for unified toolchains and the difficulty of manipulating log-based scenarios at scale.
• Reinforcement Learning (RL) Based (4 papers): Methods like MARL-OT use multi-agent RL to coordinate surrounding vehicles to trigger failures.
  • Practitioner Feedback: Aligns with the shift to reactive, closed-loop testing. Concerns were raised about the interpretability of generated scenarios and whether they remain realistic (plausible) versus just mathematically optimal.
• Generative Model-Based (2 papers): Methods like CADiffusion and SafeMVDrive use diffusion models and Vision-Language Models (VLMs) to generate adversarial images or multi-view videos.
  • Practitioner Feedback: Novel and valuable for isolating sensor-specific effects (e.g., vision-only failures). However, the complexity of integrating multi-stage pipelines into existing engineering workflows is a significant barrier.
• Adversarial Perturbation-Based (2 papers): Methods like UniAda apply pixel-level perturbations to images to induce failures.
  • Practitioner Feedback: Questioned regarding physical realism. While effective in simulation, practitioners doubted whether pixel-level variations represent real-world sensor inputs (e.g., camera noise, lighting changes).

General Finding for Q7: Practitioners value approaches that incorporate real-world data and enforce constraints to ensure scenario realism. The "curse of dimensionality" and the lack of a unified framework to compare these methods are major adoption hurdles.

B. Analysis of Q8: Completeness of E2E ADS Testing

Only one study (P17, SAFR-AV) was found addressing the completeness of E2E testing.

• Approach: A scenario-based, data-driven method using behavioral competencies and joint parameter distribution learning to maximize coverage.
• Practitioner Feedback: Considered highly relevant as it aligns with industry's move toward coverage-driven testing using real-world data.
• Remaining Challenges: Fundamental questions remain unresolved: How to define appropriate coverage targets? How to quantify coverage in high-dimensional spaces? How to link coverage metrics to safety assurance?

5. Significance and Implications

• Bridging the Research-Practice Gap: This study demonstrates that while academic research offers promising tools (especially in scenario generation), there is a disconnect in context-awareness. Research often ignores industrial constraints like toolchain integration, data quality, and the need for explainable results.
• Validation of E2E Testing Needs: The study confirms that the industry's shift to E2E architectures has created an urgent, unmet need for new testing methodologies that can handle the "black box" nature of these systems.
• Methodological Contribution: It validates the Interactive Rapid Review as a powerful method for software engineering research, enabling rapid, evidence-based decision-making and fostering sustained academia-industry collaboration.
• Future Directions: The authors call for:
  1. Development of unified frameworks to evaluate and compare testing methods.
  2. Research focused on realism and rationality of generated scenarios (moving beyond simulation-only metrics).
  3. Solutions for the sim-to-real gap, ensuring that adversarial or generated scenarios reflect physical sensor limitations.
  4. Further research on testing completeness metrics specifically for E2E systems.

In conclusion, the paper argues that for ADS testing research to be impactful, it must evolve from purely algorithmic novelty to context-aware, industry-relevant solutions that address the practical realities of automotive engineering.