Req2Road: A GenAI Pipeline for SDV Test Artifact Generation and On-Vehicle Execution

1. Problem Statement

The transition to Software-Defined Vehicles (SDVs) has drastically increased software complexity, leading to a surge in testing volume and variability. Current automotive testing practices face several critical challenges:

• Heterogeneous Inputs: Requirements are often written in natural language, while specifications include mixed media (text, tables, diagrams), making automated extraction difficult.
• Fragmented Toolchains: Test assets are scattered across different tools, leading to interoperability gaps.
• Manual Effort & Traceability: Test planning remains largely manual, inconsistent, and poorly aligned with evolving requirements. Traceability between natural language requirements and executable test code is weak.
• Signal Hard-coding: Traditional test scripts often rely on hard-coded signal names, reducing portability across different subsystems and test benches.

2. Methodology: The Req2Road Pipeline

The authors propose Req2Road, a Generative AI (GenAI) pipeline designed to transform heterogeneous specification artifacts into executable test scenarios and scripts. The pipeline integrates Large Language Models (LLMs), Vision-Language Models (VLMs), and Retrieval-Augmented Generation (RAG).

The workflow consists of four main phases:

A. Input Processing & Signal Extraction

• Inputs: Project textual requirements, behavioral diagrams (e.g., UML state/sequence diagrams), and the Vehicle Signal Specification (VSS) catalog.
• RAG-based Signal Mapping: To prevent hallucinations, the system does not prompt the LLM against the entire VSS catalog (which can contain thousands of signals). Instead, it uses a RAG setup:
  1. Embeds the scenario text/diagram and VSS entries using a SentenceTransformer.
  2. Performs nearest-neighbor search to retrieve a small candidate set (e.g., top 16 signals).
  3. Re-ranks candidates using a cross-encoder.
  4. The LLM selects the final relevant signals from this shortlist, ensuring semantic grounding.

B. Gherkin Scenario Generation

• Phase 1 (Preliminary): LLMs and VLMs generate initial Gherkin (BDD) scenarios from natural language requirements and flowcharts.
• Phase 2 (Refinement): These scenarios are mapped to the specific VSS signal paths identified in the previous step. This creates a "VSS-enriched" Gherkin specification that bridges high-level logic with low-level vehicle signals.
• Human-in-the-Loop: A human reviewer validates the generated scenarios for semantic correctness, thresholds, and intent before proceeding.

C. Test Script Generation

• The refined Gherkin scenarios are converted into executable Python test scripts using the Behave framework.
• The LLM is prompted with the Gherkin scenario, examples of the KUKSA client API (for VSS access), and Behave framework templates.
• The output includes environment.py and steps/*.py files, which interact with the vehicle signals via standardized APIs.

D. Execution Environments

The pipeline is evaluated in two environments:

1. Software-in-the-Loop (SiL): Using the digital.auto playground (a cloud-based, containerized SDV runtime).
2. Vehicle-in-the-Loop (ViL): Executing the same generated artifacts on a physical vehicle equipped with an NVIDIA Jetson acting as a VSS broker.

3. Key Contributions

1. End-to-End Architecture: A demonstration of a fully automated pipeline translating natural language requirements and diagrams into runnable test code for SDVs.
2. VSS Integration: The use of VSS as a standard abstraction layer allows test artifacts to be portable across different hardware configurations and test benches (SiL to ViL) without rewriting code.
3. RAG for Signal Mapping: A novel approach to using RAG to constrain LLMs to valid VSS signals, significantly reducing hallucination rates compared to prompting against full catalogs.
4. Multi-Modal Processing: Effective use of VLMs to extract logic from UML diagrams and LLMs for text processing, handling the mixed-media nature of automotive specifications.
5. Real-World Validation: Successful execution of GenAI-generated tests on a physical vehicle (Child Presence Detection System), proving the feasibility of "Requirements-to-Road."

4. Experimental Results

The system was evaluated using a Child Presence Detection System (CPDS) case study involving 36 natural language requirements.

• Gherkin Generation:
  • 32 out of 36 requirements (89%) were successfully transformed into executable Gherkin scenarios without modification.
  • The remaining 4 required minor human adjustments, primarily for ambiguous edge cases or complex "OR" conditions.
• VSS Signal Mapping:
  • RAG Effectiveness: Using a pre-filtered shortlist of 16 signals yielded perfect recall (4/4 correct) with 0 false positives for GPT-4o-mini.
  • Full Catalog Failure: When prompted against the full catalog of 982 signals without RAG, performance degraded significantly (2/4 correct for GPT-4o-mini, 19 false positives for local Vicuna models).
• Code Generation:
  • For single or dual requirements, models achieved a pass@3 of 1.0 (all generated solutions were correct).
  • For complex scenarios combining three requirements, only GPT-4.1 achieved pass@3 = 1.0; other models failed due to syntax errors or semantic logic violations.
• Execution:
  • Generated tests passed successfully in both the digital.auto SiL environment and on the physical vehicle (ViL).
  • The ViL test successfully triggered HVAC adjustments, door unlocking, and light activation based on the generated logic.

5. Significance and Future Work

Significance:

• Feasibility Demonstration: The paper proves that GenAI can bridge the gap between unstructured requirements and executable safety-critical tests in an SDV context.
• Portability: By decoupling test logic from proprietary ECU interfaces via VSS, the approach enables significant reuse of test assets across different vehicle platforms.
• Efficiency: While human review is still necessary, the pipeline automates the initial drafting, signal mapping, and code scaffolding, reducing the time from requirement to test execution.

Limitations & Future Directions:

• Complexity Sensitivity: Code generation correctness drops as scenario complexity (number of combined requirements) increases.
• Hallucination Control: RAG and pre-filtering are mandatory; direct prompting on large catalogs is unreliable.
• Safety Assurance: The paper notes that while artifacts are executable, their effectiveness (fault detection capability) needs further validation via mutation testing.
• Governance: For safety-critical systems, strict guardrails (schema validation, human approval gates) are required before generated tests can influence release decisions.

In conclusion, Req2Road represents a significant step toward automating the verification of Software-Defined Vehicles, offering a scalable, portable, and semi-automated approach to test generation that leverages the latest advancements in GenAI while maintaining necessary human oversight.