Technical Summary: LLM-Driven Cost-Effective Requirements Change Impact Analysis

Problem Statement

Software requirements are inherently subject to change throughout the development lifecycle. Identifying the impact of these changes on other requirements (Change Impact Analysis, or CIA) is critical for maintaining system consistency, safety, and compliance. However, manual identification is error-prone, effort-intensive, and often fails to capture implicit or semantic dependencies, particularly in large, complex systems. Traditional automated approaches, such as traceability link recovery (TLR) or syntactic dependency analysis, often struggle with the variability of change requests, which may be expressed as descriptive natural language rationales rather than direct requirement modifications. Existing TLR techniques, designed to recover links between artifacts using similar natural language styles, frequently fail to capture the nuanced, indirect, and semantic dependencies required to accurately predict change propagation when a change rationale is provided.

Methodology: ProReFiCIA

The authors propose ProReFiCIA (Prompt-Refinement-Filtering for CIA), an automated, LLM-driven approach designed to identify impacted requirements when a change rationale is introduced. The methodology is structured into two main stages:

Stage 1: Optimization of Model and Prompt

Before applying the full pipeline, the authors conducted an extensive evaluation to determine the most robust Large Language Model (LLM) and prompt configuration.

• Prompt Engineering: Using the RICE template (Role, Instruction, Context, Examples), the authors designed 64 distinct prompt variations. These variations systematically included or excluded specific details regarding the model's persona, instructions on handling indirect dependencies (e.g., commonsense reasoning, semantic relations), and context about the CIA task and domain.
• Model Selection: Five leading LLMs were evaluated: GPT-4o, DeepSeek, LLaMa, Mistral, and Gemini.
• Execution Strategy: The approach utilizes Cache-Augmented Generation (CAG), where the full list of requirements is provided to the LLM in a single context window to reduce query counts and leverage shared context, rather than iterative querying.

Stage 2: The ProReFiCIA Pipeline

The core approach consists of three sequential steps applied to the optimal model-prompt combination:

1. Initial Execution: The LLM processes the change rationale and the full requirement list to generate an initial set of impacted requirements.
2. Refinement Step: To mitigate the "lost-in-the-middle" problem (where models overlook items in long contexts), the LLM is prompted again with the remaining unselected requirements. This iterative check aims to recover missed true positives.
3. Filtering Step: To reduce false positives and minimize analyst effort, a two-phase filtering algorithm is applied:
   • Ranking: The LLM re-ranks the candidate requirements based on confidence and logical relevance.
   • Selection: A fine-tuned Natural Language Inference (NLI) model (specifically nli-deberta-v3-large) evaluates whether the selected requirement and its justification logically entail the change rationale. The algorithm retains requirements if they are entailed by the NLI model or if they appear in the top 50% of the ranked list (preserving high-confidence candidates even if the NLI model is uncertain).

Integration of Domain Knowledge (RAG)

To address scenarios where domain-specific knowledge is required to understand implicit dependencies (e.g., understanding "EMC guidelines" without explicit mention), the authors integrated Retrieval-Augmented Generation (RAG). Relevant domain concepts (sourced from Wikipedia) are retrieved and injected into the prompt context to enhance the model's understanding.

Key Contributions

1. Systematic Prompt Analysis: The design and evaluation of 64 prompt variations to analyze how prompt detail and structure influence LLM performance on the CIA task across different domains and complexity levels.
2. Comprehensive Model Evaluation: A comparative assessment of five distinct LLMs to identify the most robust models for this specific task.
3. ProReFiCIA Framework: An automated approach combining refinement and filtering stages with advanced prompt engineering to enhance performance while minimizing the need for labeled training data.
4. Empirical Validation: Evaluation on unseen industrial datasets demonstrating high cost-effectiveness.
5. RAG Enhancement: Demonstration that incorporating domain knowledge via RAG significantly boosts recall with minimal cost increase.

Results

The approach was evaluated on two datasets: WASP (mobile service platform) and SAT-DLink (satellite data link management system), with a specific focus on the unseen, complex SAT-DLink-Test subset.

• Model Robustness: GPT-4o and LLaMa demonstrated the highest stability and accuracy across different prompt variations and datasets. GPT-4o showed the most consistent behavior regarding prompt details.
• Optimal Configuration: The combination of GPT-4o with prompt P53 (a specific configuration of the RICE template) was identified as the best model-prompt pair.
• Performance Metrics (SAT-DLink-Test):
  • Baseline (GPT-4o-P53 without post-processing): 73.3% recall, 1.5% cost (requirements reviewed).
  • ProReFiCIA (Refinement + Filtering): Achieved 85.7% recall while requiring the analyst to review only 3.0% of the total requirements.
  • ProReFiCIA with RAG: Incorporating domain knowledge increased recall to 95.8% with a slight cost increase to 3.4%.
• Comparison to Baselines: ProReFiCIA significantly outperformed baselines, including a RAG-based Chain-of-Thought (CoT) approach and semantic similarity techniques (using LLM embeddings with various cutoff strategies). For instance, it achieved 34 percentage points higher effectiveness than the best semantic similarity baseline (ST2) and 96 percentage points lower cost than the fixed-threshold baseline (ST1).
• Scalability: The method exhibits approximately linear growth in cost and runtime with the number of requirements, remaining computationally stable.

Significance and Claims

The paper claims that ProReFiCIA offers a cost-effective solution for requirements change impact analysis. By leveraging the reasoning capabilities of LLMs through careful prompt engineering and post-processing, the approach successfully identifies the vast majority of impacted requirements (up to 95.8% with RAG) while drastically reducing the human review burden (to ~3% of the total requirement set).

The authors emphasize that this approach addresses the limitations of traditional TLR techniques by handling complex, natural language change rationales and capturing implicit semantic dependencies. They note that the method requires only a small amount of labeled data (for the NLI fine-tuning) and is adaptable across domains. The study concludes that providing the full context of requirements to the LLM (via CAG) combined with refinement and filtering is superior to iterative querying or retrieval-based candidate selection for this specific task, as it allows the model to leverage shared context for more informed decisions.

The paper modestly acknowledges that while the results are promising, further validation on larger and more diverse datasets is necessary to strengthen external validity, and future work will focus on mitigating false positives and adapting the framework to other artifacts like source code.