Early-Stage Product Line Validation Using LLMs: A Study on Semi-Formal Blueprint Analysis

1. Problem Statement

Software Product Line Engineering (SPLE) relies on Scoping as a critical early phase where engineers define product boundaries, candidate features, and variability assumptions. Currently, validation of these decisions occurs late in the process, only after a formal Feature Model (FM) is constructed using a formal language (e.g., UVL).

• The Gap: This delay means that inconsistencies, dead features, or infeasible variability assumptions are often discovered only during requirements engineering or implementation, leading to costly rework.
• The Challenge: Existing automated analysis operations (AOs) require formal models and SAT/CSP solvers. There is no established method to perform rigorous logical validation on the semi-formal, textual blueprints used during the initial scoping phase.
• The Question: Can Large Language Models (LLMs), acting as lightweight inference engines, perform feature model analysis operations directly on these semi-formal textual descriptions without converting them to formal code first?

2. Methodology

The authors propose an Early Validation Workflow and conduct a large-scale empirical evaluation to test its feasibility.

A. The Workflow

1. Blueprint Creation: Domain experts express variability assumptions in a semi-formal textual blueprint. This is a constrained natural language format (e.g., "Feature A requires Feature B," "Feature C excludes Feature D") that bridges the gap between free-text notes and formal UVL models.
2. LLM-Based Analysis: The blueprint is fed to an LLM with a tailored prompt to execute specific Analysis Operations (AOs).
3. Feedback Loop: The LLM returns actionable findings (e.g., dead features, satisfiability status) to the expert for immediate refinement.

B. Experimental Setup

• Models Evaluated: 12 state-of-the-art LLMs, categorized into:
  • General-purpose: GPT-4.1, Claude Sonnet 4, DeepSeek Chat, Llama 4 Scout, Grok 4 Non-Reasoning.
  • Reasoning-optimized: Grok 4 Fast Reasoning, GPT-5 mini, Gemini 2.5 Pro, Claude Sonnet 4 Thinking, DeepSeek Reasoner, Llama 4 Maverick, Gemini 2.5 Flash.
• Dataset: 10 feature models derived from the UVL-Hub and Feature-Model-Benchmark repositories, ranging from toy models (6 features) to large industrial-scale models (6,867 features). These were converted into semi-formal blueprints.
• Analysis Operations (AOs): 16 distinct AOs were tested, divided into:
  • Solver-free: Structural metrics (e.g., feature counts, tree depth).
  • Solver-based: Semantic reasoning (e.g., satisfiability, dead feature detection, configuration counting, generalization).
• Oracle: FLAMA (using the Glucose3 SAT solver) was used as the ground-truth oracle to verify LLM outputs.
• Evaluation Metrics: Accuracy (exact match with FLAMA), Runtime, Token Usage, and Error Taxonomy (Unparseable, Format-correct but wrong, Truncated, Hallucinated).

3. Key Contributions

1. Workflow Formalization: Defined a novel workflow integrating semi-formal blueprints with LLM-based analysis to enable validation before formal modeling.
2. Large-Scale Empirical Study: The first systematic evaluation of 12 LLMs across 16 AOs and 10 blueprints of varying complexity, comparing them against a solver-based oracle.
3. Error & Cost Analysis: Identified specific failure modes (semantic slips, propagation errors) and analyzed the accuracy-cost trade-offs, providing guidelines for model selection in SPLE scoping.

4. Key Results

A. Accuracy (RQ1)

• Reasoning-Optimized Models Dominate: Reasoning-optimized LLMs achieved an average accuracy of 88–89% (e.g., Grok 4 Fast Reasoning at 89.7%, GPT-5 mini at 88.9%, Gemini 2.5 Pro at 88.2%), significantly outperforming general-purpose models (avg. 61.0%).
• Task Dependency:
  • Verification tasks (e.g., "Is this satisfiable?") were handled well by top models (>85%).
  • Structural counting tasks (e.g., counting mandatory features) remained difficult, with accuracy dropping to ~58% even for top models due to semantic misunderstandings (e.g., confusing "A or B" with "A and B").
• Complexity Sensitivity: Accuracy decreased steadily as blueprint complexity (feature count, depth, constraint density) increased. No model achieved 100% accuracy on the largest, most complex models (CNNl, CNNf).

B. Efficiency & Cost (RQ2)

• Trade-offs: Reasoning-optimized models are significantly more expensive in terms of time and tokens (2,400–9,500 seconds runtime; 500K–840K tokens) compared to general-purpose models.
• Best Trade-off: Gemini 2.5 Pro offered the best balance, achieving 88.2% accuracy with the lowest runtime among reasoning models (~2,400s). DeepSeek Reasoner was the slowest (9,500s) with lower accuracy (82.8%).
• Solver Baseline: FLAMA (the solver) was instantaneous (18s) but cannot process semi-formal blueprints directly, highlighting the LLM's role as a necessary bridge.

C. Failure Modes (RQ3)

Three primary error types were identified:

1. Semantic Slips: Misinterpreting logical constraints (e.g., treating an "OR" group as "AND" or miscounting mandatory features).
2. Incomplete Propagation: Failure to trace transitive constraints in complex models, leading to missed dead features or incorrect configuration counts.
3. Context/Output Limits: Truncation of outputs on very large blueprints (e.g., CNNf), causing incomplete analysis.

5. Significance and Implications

• Feasibility of Early Validation: The study proves that LLMs can serve as lightweight assistants for early variability validation, providing near-solver accuracy (88–89%) without the need for formal model construction.
• Practical Application: For SPLE scoping, where speed of iteration is more critical than absolute perfection, reasoning-optimized LLMs provide actionable feedback (e.g., "This feature is dead because of constraint X") that helps stakeholders refine scope decisions immediately.
• Recommendations:
  • Use ensemble strategies (majority voting across top models) to mitigate individual model weaknesses.
  • Implement disambiguation rules in prompts to reduce semantic slips in counting tasks.
  • Human oversight remains necessary, as approximately 1 in 10 analyses may still contain errors.
• Future Direction: The authors suggest extending this workflow to include more complex AOs (e.g., diagnosis, redundancy) and developing "learned routers" to select the optimal model for specific blueprint complexities.

Conclusion: While LLMs are not yet a replacement for formal solvers in final verification, they are a viable, high-value tool for early-stage product line scoping, enabling automated, explainable, and iterative validation of variability decisions before significant engineering effort is invested.