💻 computer science

TVR: Automotive System Requirement Traceability Validation and Recovery Through Retrieval-Augmented Generation

This paper introduces TVR, a Retrieval-Augmented Generation (RAG) approach leveraging Large Language Models to validate existing and recover missing traceability links between stakeholder and system requirements in automotive software development, demonstrating high accuracy and practical effectiveness in industrial settings.

Original authors: Feifei Niu, Rongqi Pan, Lionel C. Briand, Hanyang Hu

Published 2026-07-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Feifei Niu, Rongqi Pan, Lionel C. Briand, Hanyang Hu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a massive, high-tech car. This car isn't just metal and rubber; it's a complex brain made of thousands of software instructions. To make sure this brain works safely, engineers write two types of "to-do lists":

The "Big Picture" List (Stakeholder Requirements): Written by the people who want the car to work (drivers, regulators, mechanics). It says things like, "If the car loses contact with the engine computer, we need a warning light."
The "Technical Blueprint" List (System Requirements): Written by the engineers who actually build the software. It says things like, "If MESSAGE_1 is missing for 5 cycles, turn on DTC_001."

The Problem: The Broken Chain
In a perfect world, every item on the "Big Picture" list would be perfectly matched to a specific item on the "Technical Blueprint" list. This matching is called traceability. It's like a chain of paperclips connecting the two lists.

However, in the real world, this chain often breaks.

Sometimes, a change is made to the "Big Picture" list, but the engineers forget to update the "Technical Blueprint."
Sometimes, a human makes a mistake and connects the wrong items.
Sometimes, items are missing entirely.

If these chains are broken, the car might not warn the driver when something is wrong, or it might warn them when nothing is wrong. This is dangerous and expensive to fix later.

The Old Way vs. The New Way
Previously, engineers had to manually check every single paperclip connection. It was slow, boring, and prone to human error.

Some computer programs tried to help by looking for similar words. If the "Big Picture" list said "engine" and the "Blueprint" said "engine," they connected them. But this is like trying to match a recipe to a grocery list just by looking for the word "flour." It's too simple. The technical lists use very specific, jargon-heavy language that looks different from the big-picture lists, even when they mean the same thing.

The Solution: TVR (The Smart Matchmaker)
The paper introduces a new tool called TVR (Traceability Validation and Recovery). Think of TVR as a super-smart, experienced librarian who uses a special technique called RAG (Retrieval-Augmented Generation).

Here is how TVR works, using a simple analogy:

1. The "Show Me, Don't Just Tell Me" Approach

Imagine you are teaching a new intern how to spot a fake ID.

The Old Way (Zero-Shot): You just say, "Here is an ID. Tell me if it's real or fake." The intern guesses and gets it wrong because they don't know the rules.
The New Way (TVR/RAG): Before asking the intern to judge the new ID, you pull out a stack of past examples from a filing cabinet. You say, "Look at these 3 examples where we said 'Real' and these 3 where we said 'Fake'. Notice how the font looks in the 'Real' ones? Now, look at this new ID. Based on those examples, is it real or fake?"

TVR does exactly this with Large Language Models (AI). Instead of just asking the AI "Is this link correct?", it first retrieves similar examples of correct and incorrect links from the company's history. It shows these examples to the AI to "teach" it the specific patterns of this car company before it makes a decision.

2. What TVR Actually Does

The paper tested TVR on real automotive data involving Diagnostic Trouble Codes (DTCs)—these are the error codes that pop up on your dashboard (like "Check Engine").

Validation (Checking the Chain): TVR looks at existing connections between the "Big Picture" and "Technical" lists and says, "Yes, this link is correct," or "No, this link is broken."
- Result: It got this right 98.87% of the time.
Recovery (Finding Missing Links): TVR looks at items that aren't connected yet and asks, "Hey, these two actually belong together, even though no one linked them yet."
- Result: It found missing links with 85.50% accuracy.
Robustness (Handling New Styles): Sometimes engineers write requirements in slightly different ways (like using different words for the same thing). TVR was tested on these "unseen" variations and still got it right 97.13% of the time.

3. Why This Matters

The paper claims that TVR is a practical tool for the automotive industry.

It saves time by automating the tedious checking of thousands of connections.
It explains why it made a decision (e.g., "I said 'No' because the technical list didn't mention the specific message the big picture list asked for").
It handles the messy, jargon-filled reality of car software better than older methods.

In Summary
The paper presents TVR as a highly accurate, AI-powered assistant that acts like a seasoned expert. Instead of guessing, it looks at past examples of what "correct" and "incorrect" connections look like in the automotive world, then uses that knowledge to fix broken chains and find missing ones in car software requirements. The result is safer, more consistent, and less error-prone car software.

Technical Summary: TVR – Traceability Validation and Recovery Through Retrieval-Augmented Generation

1. Problem Definition

In automotive software development, maintaining accurate traceability between stakeholder requirements (high-level needs) and system requirements (detailed specifications) is critical for ensuring consistency, correctness, and regulatory compliance (e.g., ASPICE, ISO 26262). However, manual traceability processes are prone to errors, leading to:

Improper propagation of changes: Updates to stakeholder requirements often fail to reflect in system requirements, creating outdated or invalid links.
Human mapping errors: Incorrect mappings or missing links arise due to the labor-intensive nature of manual verification.
Linguistic and structural variations: Requirements are expressed with varying terminology, abstraction levels, and templates (e.g., "Lost Communication" vs. "Implausible Data" Diagnostic Trouble Codes or DTCs). Traditional similarity-based approaches (e.g., TF-IDF, SentenceBERT) and standard Large Language Model (LLM) prompting (Zero-Shot, Chain-of-Thought) struggle to generalize across these heterogeneous variations and often fail to distinguish subtle semantic differences in message and signal definitions.

The specific challenge addressed is the validation of existing traceability links and the recovery of missing links between DTC stakeholder and system requirements, where a system requirement may cover multiple stakeholder requirements, and textual similarity alone is insufficient for validation.

2. Methodology: The TVR Approach

The authors propose TVR (Traceability Validation and Recovery), a framework leveraging Retrieval-Augmented Generation (RAG) enhanced with In-Context Learning. Unlike traditional RAG systems that retrieve external documents to augment domain knowledge, TVR retrieves previously labeled requirement pairs to serve as reasoning examples.

Core Components

Retriever:
- Input: A test pair $\langle \text{stakeReq}, \text{sysReq} \rangle$ .
- Mechanism: Generates embeddings for concatenated requirement pairs using the Amazon Titan model (selected due to industry partner constraints).
- Selection: Retrieves the top- $k$ most similar valid (positive) and top- $k$ most similar invalid (negative) examples from a training database ( $D_{train}$ ).
- Similarity: Uses Cosine Similarity (optimized at $k=3$ ) via the FAISS library.
Generator:
- Model: Primarily evaluated using Claude 3.5 Sonnet, though 13 LLMs were tested.
- Prompting Strategy: The generator receives a prompt containing:
  - Instruction: A specific task to verify if the system requirement covers the specific message or signal in the stakeholder requirement (narrowing the scope from general traceability to specific signal coverage).
  - Context: The $k \times 2$ retrieved examples (valid and invalid) to guide the model's reasoning.
  - Input: The test pair to be evaluated.
  - Output Indicator: A binary "Yes" or "No" response.
- Goal: The LLM uses the retrieved examples as "demonstrations" to learn the decision patterns for traceability correctness, rather than relying on external knowledge retrieval.
Traceability Recovery (Preprocessing):
To recover missing links without an exhaustive Cartesian product search, TVR employs a three-step filtering strategy:
- Variation Matching: Pairs requirements based on DTC type (e.g., "Lost Communication" only links to "Lost Communication").
- Condition Matching: Pairs requirements based on the condition type (Mature vs. Demature).
- Message Overlap Matching: Filters pairs that do not share at least one common message or signal token.
- The filtered candidates are then validated by TVR to identify missing valid links.

3. Key Contributions

TVR Framework: A novel RAG-based approach specifically designed for validating and recovering traceability links in automotive systems, utilizing in-context examples rather than external knowledge retrieval.
Robustness to Variations: Demonstrated ability to handle unseen requirement variations (different templates and terminologies) by leveraging the semantic understanding of LLMs augmented with representative examples.
Comprehensive Evaluation: A rigorous evaluation across 13 LLMs (including Claude, Llama, Mistral, and Titan families) and four prompting strategies (Zero-Shot, CoT, Few-Shot, Self-Consistency), comparing TVR against state-of-the-art baselines (LiSSA, TF-IDF, SentenceBERT).
Industrial Validation: Evaluation on a real-world industrial dataset of 2,132 DTC requirement pairs from an automotive partner, with ground truth established through structured annotation by system engineers.

4. Experimental Results

The study was conducted on a dataset of 1,320 stakeholder requirements linked to 48 system requirements (2,132 links).

Traceability Validation:
- Accuracy: TVR (using Claude 3.5 Sonnet with RAG) achieved 98.87% accuracy.
- Macro-F1: Achieved 96.72%, significantly outperforming the best baseline (LiSSA-CoT at 52.89% and SentenceBERT at 57.82%).
- Comparison: Zero-Shot prompting on the same model yielded only 79.55% accuracy, highlighting the necessity of the RAG/In-Context approach.
- Robustness: When tested on unseen requirement variations (cross-validation across four variation categories), TVR maintained 97.13% accuracy and a Macro-F1 of 90.86%.
Traceability Recovery:
- After the three-step preprocessing reduced the candidate pool from ~30k to 1,919 pairs, TVR identified 502 potential missing links.
- Correctness: Human verification confirmed 85.50% of the recovered links were correct.
Error Analysis:
- Errors primarily occurred when the LLM focused on trigger conditions rather than the target input message, or when it considered components beyond the specific signal (e.g., validation rules).
- TVR and retrieval-based baselines (SentenceBERT) showed orthogonal error patterns, suggesting potential for hybrid approaches.

5. Significance and Claims

The paper claims that TVR offers a practical, high-accuracy solution for industrial settings where requirements traceability is critical but manually maintained.

Efficiency: It significantly reduces the manual effort required for traceability verification, addressing the time and cost constraints reported by industry partners.
Reliability: The high accuracy (98.87%) and robustness to unseen variations make it suitable for deployment in complex, evolving automotive systems.
Interpretability: The framework supports explainability; when configured to do so, the LLM can provide step-by-step reasoning (e.g., identifying the specific message and checking for its presence in the system requirement), aiding engineers in verifying automated decisions.
Limitations & Future Work: The authors acknowledge that the approach relies on the quality of the retrieval database and that false positives can occur due to the model's occasional failure to strictly adhere to the "message-only" focus. They suggest future work could involve hybridizing retrieval-based similarity with LLM reasoning and refining preprocessing to exclude trigger conditions.

The authors conclude that while TVR is currently tailored to automotive DTC requirements, its underlying principles of using retrieval-augmented in-context learning for requirement validation are adaptable to other domains requiring high-level to low-level requirement traceability.